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ARMY  RESERVE  COMPONENTS  RESEARCH  ROADMAP 
(VOLUME  2):  RESEARCH  AGENDA 


INTRODUCTION 

The  purpose  of  this  project  is  twofold:  (a)  to  provide  in  a  single  source  document  the 
results  of  selected  research  and  development  (R&D)  efforts  undertaken  by  the  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  on  behalf  of  the  U.S.  Army’s 
Reserve  Components  (RC)  (i.e.,  Army  National  Guard  [ARNG]  and  U.S.  Army  Reserve 
[USAR]),  and  (b)  to  generate  an  agenda  for  future  ARI  work  that  is  responsive  to  current  and 
projected  RC  needs.  Project  results  are  documented  in  a  two-volume  report.  The  first  volume 
describes  the  R&D  results  obtained  in  the  areas  of  manpower,  personnel,  and  training  from  the 
year  1980  to  the  present  (Ford,  Schultz,  Knapp,  &  Katz,  1997). 

The  purpose  of  the  research  agenda  described  in  this  second  volume  is  to  provide  ARI 
researchers  and  Army  policy-makers  with  suggestions  about  how  scarce  research  resources  might 
be  most  effectively  spent  to  support  the  needs  of  the  RC.  Critical  research  questions  across  a 
variety  of  manpower,  personnel,  and  training  domains  are  identified  to  help  assure  that  they  are 
not  neglected  in  ARI  research  program  planning  activities. 

Before  turning  to  discussions  of  individual  areas  of  research,  the  following  sections  (a) 
describe  the  study  approach,  (b)  review  the  nature  of  the  RC  environment,  (c)  discuss  ARI’s 
research  and  consulting  roles  in  support  of  the  RC,  and  (d)  describe  the  organization  of  the 
remainder  of  the  report. 


Study  Approach 

As  mentioned,  one  purpose  of  this  project  was  to  identify  and  describe  prior  ARI  research 
supporting  the  RC.  The  Volume  1  report  (Ford  et  al.,  1997)  briefly  describes  the  RC 
environment  and  how  this  environment  differs  from  that  of  the  Active  Component  (AC).  The 
bulk  of  the  report  reviews  training-  manpower-  and  personnel-related  findings/products 
developed  by  ARI  since  1980.  These  “product  summaries”  cover  why,  how,  and  with  whom  the 
research  was  conducted;  what  was  found/developed;  what  the  implications  are;  and  where  more 
information  can  be  found.  The  latter  is  provided  in  the  form  of  specific  product  and  related 
references  placed  at  the  end  of  each  summary. 

Development  of  a  research  agenda  was  the  second  major  goal  of  this  project.  The  agenda 
is  based  on  the  integration  of  input  from  a  variety  of  sources,  including  interviews  with  policy¬ 
makers  and  researchers,  ARI  research  literature,  and  articles  and  documents  describing  the  RC 
environment  and  related  research  conducted  outside  ARI. 

A  critical  component  of  this  project  was  the  set  of  interviews  that  were  conducted  to 
gather  information  and  ideas  from  researchers  and  RC  personnel.  Interviews  were  conducted 
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from  October  1996  through  February  1997.  While  most  interviews  were  conducted  face-to-face, 
some  were  carried  out  via  telephone. 

A  smnmary  of  the  individuals  who  participated  in  the  interviews  is  provided  in  Appendix 
A.  Our  goal  was  to  interview  researchers  and  RC  personnel  (both  ARNG  and  USAR)  representing 
a  variety  of  perspectives,  including  those  most  familiar  with  training,  manpower,  or  personnel 
issues  and  individuals  at  higher  levels  who  are  responsible  for  research  and  operations  that  span  all 
of  these  areas.  The  focus  of  the  interviews  varied  somewhat  depending  upon  the  interviewee.  All 
participants  were  asked  to  describe  trends  affecting  the  RC  that  might  affect  the  type  of  research 
that  would  be  useful  for  addressing  future  needs.  Researchers  were  asked  to  describe  their  research 
and  ideas  for  future  research;  RC  personnel  were  asked  to  indicate  what  types  of  research  questions 
would  be  most  relevant  to  them. 

Information  and  ideas  generated  in  the  interviews  have  been  incorporated  into  the  content 
of  this  report.  Indeed,  the  interviews  were  instrumental  in  determining  both  the  structure  and 
content  of  the  following  discussions  of  training,  manpower,  and  personnel  research  provided 
herein.  Unedited  summaries  of  the  interviews  have  been  provided  to  ARI. 

Relevant  ARI  research  literature  was  located  and  reviewed  as  part  of  preparation  of  the 
project’s  Volume  1  report.  Throughout  the  course  of  the  project,  we  also  collected  articles  and 
documents  that  described  the  current  and  projected  structure  and  environments  for  the  USAR  and 
ARNG.  These  included,  for  example,  posture  statements,  the  Army  Green  Book,  General 
Accounting  Office  (GAO)  reports,  and  newspaper  clippings. 

To  gather  information  about  related  research  literature  not  sponsored  by  ARI,  inquiries 
were  made  to  the  Defense  Technical  Information  Center  (DTIC)  and  researchers  at  several  other 
institutions  were  interviewed.  The  intent  was  to  be  able  to  give  a  flavor  of  what  other  research 
may  be  available  to  ARI  and  the  RC  to  help  address  critical  RC-related  research  questions. 

Despite  initial  attempts  to  quantify  user  and  researcher  priorities,  at  least  at  a  fairly  global 
level,  it  became  increasingly  clear  that  most  interview  participants  were  not  comfortable 
recommending  priorities  that  cut  across  different  research  domains.  The  problem  was  that  most 
interviewees,  both  researchers  and  policy-makers,  felt  that  they  had  perspectives  that  were 
limited  to  certain  areas  (e.g.,  training  or  recruiting).  Therefore,  we  suspended  plans  to  have  them 
complete  a  follow-up  prioritization  survey.  Such  a  survey  was  used  successfully  in  the  Joint 
Service  classification  research  roadmap  study  (J.P.  Campbell,  Russell,  &  Knapp,  1994).  The 
focus  of  that  project,  however,  was  on  one  general  research  domain.  The  focus  of  the  present 
project  is  much  broader.  As  a  result,  we  attempted  to  identify  which  issues  were  viewed  as 
worthy  of  further  exploration  and  which  were  not,  based  on  the  qualitative  input  of  our 
interviewees  as  well  as  our  analysis  of  related  information. 
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The  RC  Environment 


The  Volume  1  report  (Ford  et  al.,  1997)  describes  how  the  RC,  both  USAR  and  ARNG, 
are  very  different  environments  from  the  active  Army.  These  differences  create  unique  needs  for 
the  RC  that  catmot  necessarily  be  addressed  by  R&D  work  for  the  active  Army.  These  needs 
revolve  around  four  themes. 

First,  duty  time  is  severely  limited  for  USAR  and  ARNG  personnel.  Most  obviously,  this 
limits  the  amount  of  training  time  available.  Speed  of  training,  in  addition  to  the  cost  of  training, 
is  supremely  important.  Training  designs  and  training  effectiveness  studies  that  do  not  consider 
training  time  are,  therefore,  not  particularly  useful  for  the  RC  environment.  Furthermore,  the 
importance  of  training  time  has  an  important  implication  for  manpower  and  persormel. 

Persormel  policies  or  programs  that  can  reduce  the  need  for  training  benefit  the  RC  by  helping  it 
manage  its  time  constraint  problem. 

The  next  two  factors  both  relate  to  geographical  dispersion.  Neither  units  nor  personnel 
are  typically  located  in  a  confined  geographic  area  like  they  are  in  the  AC.  Unit  headquarters, 
training  sites,  and  equipment  storage  locations  tend  to  be  widely  dispersed.  For  example, 
companies  within  ARNG  battalions,  on  the  average,  are  spread  over  a  150-mile  radius.  This 
obviously  makes  training  above  the  company  level  difficult  for  any  time  other  than  annual 
training  (AT).  In  addition,  company  armories  are  often  not  co-located  with  their  equipment. 

Thus,  accessibility  to  equipment  is  limited.  This  problem  is  compounded  by  the  fact  that  training 
areas  may  also  be  dispersed  from  equipment  storage  sites.  These  aspects  of  dispersion  obviously 
affect  training;  they  emphasize  the  need  to  consider  where  training  is  to  be  conducted.  Travel 
time  is  not  free,  it  is  a  debit  on  duty  time.  Therefore,  the  closer  training  opportunities  are  to  unit 
headquarters,  the  more  time  units  can  spend  productively  training. 

In  addition  to  unit  dispersion,  individuals  tend  to  be  widely  scattered  as  well.  This  is 
important  because  a  considerable  amount  of  RC  work  occurs  during  times  other  than  weekend 
multiple  unit  training  assemblies  (MUTAs)  and  AT.  Only  limited  amounts  of  fimds  are  available  to 
pay  for  individuals  to  participate  in  extra  duty  time  either  for  training  or  for  the  planning  and 
management  of  training.  The  soldiers  themselves  contribute  additional  time.  Individual  dispersion 
creates  it  own  cost  when  these  soldiers,  often  on  their  own  time,  must  travel  to  meetings  or  training 
sessions.  To  the  extent  that  this  cost  exceeds  the  capability  or  willingness  for  soldiers  to  engage  in 
extra  duty  time,  the  RC  becomes  less  programmatic  in  managing  and  delivering  training.  Policies, 
programs,  and  information  transfer  techniques  that  can  increase  the  accessibility  of  individual 
soldiers  to  each  other  and  to  training  resources  are  a  particular  benefit  for  the  RC. 

The  final  two  factors  also  evolve  from  a  single  theme  that  plays  out  at  the  individual  and 
unit  levels.  Turbulence,  as  described  by  the  Volume  1  report  (Ford  et  al.,  1997),  concerns  both  the 
rate  at  which  RC  soldiers  turnover  and  the  rate  at  which  RC  units  change  missions  and  structures. 
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Individual  turbulence  is  a  significant  fact  of  life  for  RC  units  as  new  persons,  often  without 
prior  service,  are  introduced  to  units,  and  as  established  persons  move,  change  unit  affiliations,  and 
frequently  change  military  occupational  specialties  and  branches  as  well.  Individual  turbulence  is 
essentially  where  the  training  and  manpower/personnel  domains  intersect.  Turbulence  is 
moderated  by  manpower/personnel  policies,  with  the  remaining  effects  absorbed  by  training. 

Similarly,  at  a  higher  level  of  decision-making,  fluidity  in  the  assignments  of  force 
structures  to  units  creates  wide  spread  change  which  affects  whole  units  and  not  just  proportions  of 
individuals  within  units.  For  example,  the  ARNG  is  anticipating  the  structural  changes  that  will 
cascade  down  from  the  newly  released  Quadrennial  Defense  Review  (Department  of  Defense, 

1997).  While  the  new  recommendations  do  support  continuation  of  the  previously  initiated 
enhanced  Separate  Brigade  (eSB)  program  for  selected  combat  units,  it  also  recommends  the 
conversions  of  other  combat  units  to  combat  support  and  combat  service  support  ftmctions. 

Multiple  Roles  for  ARI 

The  goal  of  the  RC  is  to  maintain  soldiers  and  units  at  a  sufficient  state  of  readiness  for 
them  to  be  responsive  to  activation  and  mobili2ation.  Individuals  and  units  must  attain  and 
maintain  the  basic  skill  levels  necessary  for  them  complete  their  skill  acquisition  during 
mobilization  training.  The  factors  described  above  (time,  unit  and  individual  dispersion,  unit  and 
individual  turbulence)  create  demands  and  boundaries  for  training  and  manpower  that  all  point 
toward  the  same  direction.  Within  the  demands  of  force  restructuring,  manpower  practices  must 
reduce  the  negative  impacts  of  turbulence  and  training  practices  must  become  as  streamlined  as 
possible. 

Solutions  to  these  manpower  and  training  practices  needs  require  a  variety  of  actions. 
Needs  must  be  clearly  articulated,  with  both  constraints  and  opportunities  described.  Potential 
solutions  need  to  be  designed,  within  realistic  time  frames  and  resources.  Alternative  solutions 
should  then  be  evaluated,  through  trial  implementation,  modeling,  or  reasoned  judgment. 
Solutions  must  be  implemented  and  adjusted  as  needed.  Finally,  ongoing  evaluations  should 
continue  to  monitor  the  efficiency  of  the  program  or  practice.  These  are  all  basic  ongoing 
organizational  problem-solving  requirements. 

ARI  can  and  should  be  involved  in  each  of  these  steps,  even  though  not  all  of  the  steps 
constitute  “research”  in  the  traditional  sense.  In  other  words,  ARI  has  a  technical  advisory  role  as 
well  as  a  research  mission.  ARI  researchers  possess  the  expertise  to  provide  valuable  insights  into 
training  and  personnel  issues  based  on  the  accumulation  of  their  experiences.  When  RC  questions 
arise  in  these  areas  (e.g.,  selection  of  one  particular  instructional  media  design  versus  another), 
prudence  may  dictate  that  expert  judgment  is  the  most  efficient  decision  tool.  At  other  times, 
exploratory  research  may  be  the  best  course  of  action.  The  difference  between  the  two 
circumstances  and  deciding  whether  to  conduct  research  or  provide  expert  advice  depends  on 
whether  there  exists  a  reasonable  body  of  knowledge  from  which  to  base  recommendations,  the 
cost  of  doing  the  research,  and  the  expected  incremental  value  of  the  research  over  expert  judgment. 
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The  purpose  of  this  report  is  to  develop  a  research  agenda.  However,  ARI  advisory  functions 
cannot  be  overlooked.  ARI  is  a  store  house  of  a  wide  range  of  technical  knowledge  which  may  be 
brought  to  bear  on  the  design  and  development  of  prototype  training  and  manpower  solutions.  Often, 
that  technical  knowledge  has  implications  for  policy.  For  example,  decisions  concerning  manpower 
strategies  may  be  made  without  sufBcient  grounding  in  human  nature,  unless  careful  attention  is  given 
to  a  diversity  of  considerations  concerning  human  abilities  and  needs.  Thus  to  be  most  effective  in 
influencing  RC  practices  in  training  and  manpower,  ARI  needs  to  continue  to  be  engaged  in  policy 
consultation,  technical  consultation,  prototype  development,  and  research. 

Organization  of  Report 

The  following  sections,  which  comprise  the  bulk  of  this  volume,  discuss  R&D  work 
projected  to  be  needed  by  the  RC  over  the  short  and  long  term.  This  discussion  is  divided  into 
the  following  research  domains: 

Training 

♦  Training  Aids,  Devices,  Simulators,  and  Simulations  (TADDS)-Based  Training 

♦  Performance  Measurement 

♦  Training  Management 

♦  Distance  Training 

Manpower  and  Personnel 

♦  Recruiting 

♦  Retention 

♦  Selection,  Classification,  and  Performance  Management 

♦  Management  Systems 

♦  Leadership 

The  discussion  of  each  research  domain  is  intended  to  be  a  relatively  brief,  but 
informative,  summary  of  the  current  state  of  knowledge  as  it  relates  to  the  Army  RC.  Many  of 
these  discussions  refer  to  work  that  has  been  done  outside  ARI  and  outside  the  Army,  but 
thorough  reviews  and  analysis  of  such  work  were  beyond  the  scope  of  this  project.  Each 
discussion  ends  with  a  list  of  critical  research  questions,  with  a  short  explanation  of  each 
requirement  listed.  Most  of  the  suggestions  are  for  applied  R&D,  extending  themes  and  building 
programmatically  from  past  efforts.  This  is  particularly  true  for  the  training  research  domain. 
ARI  has  been  more  active  in  the  training  research  domain  for  the  RC  and  therefore  the  research 
agenda  for  that  domain  tends  to  elaborate  technical  nuisances  related  to  ongoing  projects  more  so 
than  the  manpower  domain.  To  create  a  greater  sense  of  comparability  between  discussions  of 
the  two  domains,  a  special  section  is  included  in  the  discussion  of  the  training  research  domain 
that  suggests  new  research  directions. 
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Following  presentation  of  each  of  the  research  domains,  an  overall  research  agenda  is 
provided.  This  serves  as  a  summary  and  integration  of  research  questions  and  issues  that  cut 
across  the  individual  research  domains.  Finally,  the  report  summary  provides  suggestions  for 
ways  in  which  the  research  agenda  might  be  put  to  most  effective  use  by  ARI  and  its  sponsors. 

TRAINING  RESEARCH  DOMAINS 

In  1985,  ARI  established  a  research  unit  in  Boise,  ID  devoted  specifically  to  the  conduct 
of  RC-specific  R&D  activities.  Research  conducted  by  that  unit  and  others  within  ARI  has 
shown  that  innovations  in  training  are  applicable  to  the  RC  environment.  Using  distance 
learning  (DL)  for  individual  qualification  and  simulations  for  collective  tactical  training  are 
examples.  It  is  also  the  case  that  the  questions  that  were  appropriate  for  earlier  generations  of 
TADSS  continue  to  be  appropriate  for  more  advanced  TADSS,  including  questions  concerning 
cost-effective  ways  to  implement  TADSS  in  an  RC  environment.  Thus  at  a  general  level,  most 
of  the  research  questions  posed  in  the  following  discussions  could  have  been  asked  years  ago, 
and  many  of  them  were.  Continued  research  along  these  lines,  however,  will  help  show  how 
new  technology  improvements  can  increasingly  make  TADSS  and  other  enhancements  practical 
solutions  to  persistent  training  needs. 

It  should  also  be  noted  that  the  questions  presented  in  the  following  sections  focus  on 
development  as  well  as  research.  This  is  a  reflection  of  Army  needs.  As  technology  has 
advanced,  training  resources  have  declined.  One  result  has  been  a  reduction  in  the  training 
development  staff  of  the  U.S.  Army  Training  and  Doctrine  Command  (TRADOC).  ARI  has 
helped  fill  the  gap  by  developing  content  for  TADSS  and  other  innovations  in  addition  to 
evaluation  the  effectiveness  of  the  applications.  Interviews  with  ARNG  officers  strongly 
reinforced  the  idea  that  ARI  should  continue  in  that  role. 

TADSS-Based  Training 

The  effective  and  efficient  use  of  TADSS  is  vital  for  enabling  RC  units  to  achieve  and  sustain 
readiness  because  TADSS  have  the  potential  to  overcome,  at  least  partially,  environmental  constraints 
for  the  RC.  Simulations,  particularly  low-cost  part-task  simulators,  can  provide  soldiers  with  more 
ready  access  to  training  than  training  on  operational  equipment.  As  previously  discussed,  RC 
environmental  constraints  are  such  that  even  small  gains  in  training  efficiency  can  be  significant.  As  a 
result,  much  of  the  research  that  guides  acquisition  of  TADSS  and  implementation  of  TADSS-based 
training  in  the  AC  and  RC  has  been  conducted  in  the  RC  environment.  Examples  include  prediction 
of  live  fire  firom  COFT,  simulation  networking  (SIMNET)  for  collective  tactical  training,  and  Janus 
for  staff  training.  The  RC’s  particular  need  for  efficiency  makes  it  a  fertile  ground  for  continued 
research  on  ways  to  develop  and  validate  TADSS-based  training.  Research  issues  associated  with  this 
domain  are  presented  in  six  sections:  cost  effectiveness  trade-offs,  part-task  training,  fidelity 
requirements,  computer-generated  forces  (CGF),  training  support,  and  strategies. 
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Training  Efficiency  and  Cost  Effectiveness  Trade-Offs 


The  fundamental  assumption  in  procuring  any  TADSS  is  that  using  the  TADSS  will 
improve  job  performance.  Conducting  research  to  confirm  such  a  positive  relationship  is 
challenging  enough,  but  more  complex  questions  about  the  monetary  bottom  line  for  training 
might  also  be  asked  (Hoffman  &  Morrison,  1992): 

•  What  gains  in  proficiency  accrue  as  a  function  of  training  costs? 

•  Do  the  transfer  benefits  of  new  TADSS  justify  their  operational  costs? 

•  To  what  extent  can  practice  on  the  TADSS  be  substituted  for  practice  on 
operational  equipment  or  in  a  field  environment? 

•  What  is  the  optimal  mix  of  training  on  TADSS  and  operational  equipment/field 
training? 

These  questions  require  an  examination  of  the  relationships  and  trade-offs  between 
training  resources  and  performance  proficiency  to  find  the  mix  of  training  methods  that  either 
maximizes  performance  for  a  set  training  budget  or  minimizes  cost  for  a  given  training  standard. 

Hoffinan  and  Morrison  (1992)  described  minimum  and  desirable  design  attributes  for  trade¬ 
off  research.  The  first  minimum  requirement  is  that  there  should  be  variation  in  resources  under 
investigation  (e.g.,  trials  on  a  training  device),  A  desirable  attribute  is  that  the  variation  shoxild  be 
systematic,  but  since  resources  are  under  the  control  of  training  managers,  natural  variation  is  more 
likely.  The  second  minimum  requirement  is  that  the  criterion  training  method  (e.g.,  live  fire) 
should  be  repeated.  Such  repetition  allows  for  “interface”  effects— when  differences  between  the 
response  characteristics  of  TADSS  and  operational  equipment  result  in  a  short-lived  deficit  in 
criterion  performance— and  provides  cost  information  on  the  operational  equipment/environment. 
The  third  minimum  attribute  is  that  other  related  training  should  either  be  held  constant  or 
measured  (to  be  controlled  during  the  analysis).  For  example,  a  trade-off  study  of  an  armor 
gunnery  device  and  live-fire  should  at  least  document  the  extent  of  sub-caliber  exercises. 

Given  the  costs  of  conducting  trade-off  research,  it  is  not  surprising  that  such  research  in 
the  Army  (or  anywhere  else)  is  rare;  Hoffman  and  Morrison  (1992)  reported  only  three  examples, 
all  in  aviation  prior  to  1981.  With  the  introduction  of  more  sophisticated,  higher-cost  TADSS, 
the  need  to  answer  the  cost-effectiveness  questions  will  increase.  Training  of  the  Individual 
Ready  Reserve  (IRR)  is  an  especially  promising  environment  for  trade-off  research  on  TADSS, 
since  multiple  repetitions  of  training  on  the  operational  equipment  are  often  planned  as  part  of 
the  train-up  (e.g.,  repetitions  to  mastery  for  aviators)  and  there  are  few  intervening  conditions. 

Alternative  Training  Effectiveness  Questions 

In  addition  to  the  resources  and  control  required  to  conduct  empirical  trade-off  studies, 
researchers  working  with  RC  training  should  consider  whether  the  trade-off  design  is  always 
appropriate  for  the  RC  environment.  The  trade-off  design  makes  the  most  sense  when  trainers 
have  a  choice  between  training  with  a  TADSS  or  training  with  operational  equipment  in  the  field. 
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RC  trainers  may  have  to  choose  between  training  a  set  of  tasks  with  a  TADSS  or  not  training 
them  at  all.  For  example,  during  the  formative  evaluation  of  the  Janus-mediated  staff  exercise 
(JMSE),  staffs  were  asked  to  compare  JMSE  with  a  field  command  post  exercise  (CPX).  One 
unit  was  unable  to  make  the  comparison  because  they  had  not  conducted  a  field  CPX  for  several 
years.  Hoffman  and  Morrison  (1992)  expressed  the  primary  RC  issue  as  “how  to  pack  more 
practice,  by  whatever  training  method,  into  the  allotted  time.”  Less  demanding  designs  can  give 
Army  decision-makers  information  on  the  effectiveness  of  TADSS  to  defend  their  acquisition 
and  encourage  their  use  in  training.  Three  alternatives  research  designs  have  been  applied  by 
ARI  researchers  in  recent  studies  of  RC  training:  transfer-of-training,  performance  improvement, 
and  prediction. 

Transfer-of-Training.  The  transfer  design  compares  a  group  that  receives  training  on  the 
TADSS  to  a  group  with  similar  characteristics  that  receives  either  no  training  or  perhaps  an 
alternate  method  of  training.  For  example,  ARI  researchers  used  this  design  to  assess  the 
transfer  of  training  from  the  Model  Training  Program-Reserve  Component  (MTP-RC) 
maintenance  training  system  to  troubleshooting  on  the  operational  Ml  tank  (Graham,  Shlechter, 
&  Goldberg,  1986).  They  found  that  recent  graduates  of  advanced  individual  training  improved 
during  repeated  trials  on  the  MTP-RC  computer-based  instruction  and  performed  significantly 
better  on  hands-on  tests  on  the  operational  tank.  Boldovici  (1987)  has  provided  ARI  with  a 
comprehensive  treatment  of  the  methodological  strengths  and  implementation  pitfalls  for 
transfer-of-training  research.  These  types  of  studies  are  particularly  strong  for  determining 
whether  training  has  occurred.  They  do  not,  however,  provide  information  about  training 
efficiency  unless  cost  information  is  included  in  the  interpretation  of  the  results. 

Performance  Improvement.  A  simpler,  but  less  powerful,  alternate  approach  is  to 
document  whether  soldiers  learn  on  the  TADSS.  Formative  evaluations  of  TADSS  for 
individual  training  often  include  pretests  and  posttests  of  material  covered  by  the  TADSS  (e.g., 
Andre'  &  Salter,  1995;  Deterline  &  Keesling,  1995;  Keesling,  1995).  Researchers  can  also 
document  learning  for  TADSS  for  collective  training,  such  as  the  assessment  of  SIMNET  for 
platoon  and  company  missions  (Shlechter,  Bessemer,  Nessleroade,  &  Anthony,  1995).  In  that 
assessment,  ARI  collected  task  ratings  by  leaders  before  and  after  training,  structured 
Observer/Controller  (0/C)  feedback  so  that  improvement  could  be  monitored,  and  measures  of 
efficiency  (e.g.,  time  to  complete  mission).  While  this  design  approach  can  be  criticized  for  its 
unsophisticated  research  methodology,  it  has  merit  when  alternative  control  and  experimental 
training  groups  cannot  be  established,  when  researchers  cannot  be  given  control  over  training 
schedules,  and  when  training  does  not  vary  among  soldiers.  It  at  least  provides  some  indication 
of  learning,  and  Kraiger  et  al.  (1997)  have  suggested  that  appropriately  worded  questions  about 
learning  may  be  used  as  signals  for  potential  usefulness  of  training. 

Prediction.  Another  way  to  estimate  whether  a  TADSS  may  be  a  potential  as  a  substitute 
for  training  on  operational  equipment  is  to  demonstrate  the  predictive  power  of  the  TADSS. 
Work  with  armor  gunnery  (Hagman  &,  Smith,  1996)  and  rifle  marksmanship  (Hagman,  in  press) 
illustrates  that  approach.  In  those  cases,  crews  or  individuals  fired  exercises  on  a  simulator  and 
fired  for  record  on  live-fire  ranges.  The  simulators  were  the  COFT  for  armor  gunnery  and  the 
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Engagement  Skills  Trainer  (EST)  for  rifle  marksmanship.  There  was  no  training  between  the 
TADSS  and  live-fire  measures.  ARI  researchers  found  that  the  relationships  between 
performance  on  the  simulator  and  live-fire  scores  were  strong  enough  that  trainers  could  predict 
live-fire  performance  fi-om  performance  on  the  simulators.  Results  from  studies  of  prediction 
inform  decisions  about  substituting  simulator  exercises  for  live-fire  and  help  trainers  focus 
resources  on  individuals  or  crews  who  need  more  training  to  reach  qualification  levels. 

Strictly  speaking,  this  design  only  shows  a  correlation  between  the  device  and  the 
operational  equipment.  However,  the  approach  has  merit  on  three  grounds.  First,  if  a  prediction 
fails  to  show  a  correlation  between  the  device  and  performance  in  the  operational  setting,  than  the 
utility  of  the  device  for  training  is  called  into  question.  The  design  is  potentially  an  inexpensive 
way  rule  out  unsound  training  devices.  Second,  if  a  prediction  study  shows  such  a  correlation,  and 
a  performance  improvement  study  can  show  that  varying  amounts  of  practice  on  the  device  can 
improve  device  performance,  then  the  combination  of  results  suggests  that  the  device  may  have 
training  value.  Third,  even  correlational  evidence  supports  training  management  decisions.  For 
example,  Hagman  and  Smith  (1996)  illustrated  that  COFT  may  be  used  to  identify  tank  crews  who 
have  a  high  probability  of  failing  to  qualify  on  tank  gunnery  Table  VIII. 

Research  Questions 

The  preceding  discussion  suggests  two  research  questions  that  should  routinely  be  asked 
when  fielding  a  new  or  improved  TADSS.  These  relate  to  its  effectiveness  and  the  cost  savings 
it  will  yield.  While  these  are  applied  research  questions,  we  also  suggest  that  ARI  pursue  more 
basic  research  that  examines  whether  there  are  types  of  skills  that  are  more  or  less  effectively 
trained  through  the  use  of  TADSS. 

How  effective  is  a  given  TADSS  in  the  RC  environment!  At  a  minimum,  researchers 
should  establish  whether  reservists  acquire  relevant  skills  using  the  TADSS  and  that  the  TADSS 
is  compatible  with  RC  training  schedules.  If  the  research  compares  soldiers  or  units,  researchers 
should  dociunent  characteristics  of  the  soldiers  and  units  being  trained  and  intervening  training 
events.  The  sample  should  be  controlled  to  assure  adequate  variance  for  projected  analyses  and 
protect  against  systematic  differences  in  experience  or  aptitude. 

What  savings  associated  with  TADSS-based  training  should  be  expected?  Even  if  an 
empirical  trade-off  study  is  not  practical,  decision-makers  need  cost  data  to  determine  whether  to 
procure  the  TADSS  for  implementation  in  RC  training.  If  a  TADSS  is  found  to  be  effective, 
researchers  should  identify  where  savings  from  implementing  the  TADSS  would  be  reflected. 
For  example,  Hagman  and  Smith  (1996)  calculated  anununition  savings  associated  with 
increases  in  first-run  crew  qualification.  Similar  information  could  be  developed  for  costs  of 
non-gunnery  collective  training  conducted  in  the  field,  such  as  a  platoon  level  field  training 
exercise  (FTX),  to  include  numbers  of  personnel,  numbers  of  vehicles  and  radios,  miles  on 
vehicles,  hours  of  use  of  simulators,  and  dollars  for  other  costs.  Documenting  costs  of  field 
training  will  require  focused  data  collection,  since  standard  record  keeping  systems  vary  from 
installation  to  installation  (Ashworth,  Phelps,  Graham,  &  Wisher,  1992)  and  are  maintained  with 
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varying  degrees  of  accuracy  and  timeliness  (Keenan,  Keesling,  &  Graney,  1996).  Further,  units 
do  not  typically  record  resource  expenditures  on  an  event-by-event  basis. 

Can  performance  categories  be  identified  for  which  alternative  TADSS  designs  are 
most  effective?  The  research  cited  above  included  training  for  both  cognitive  and  physical  skills. 
In  some  simulations,  the  simulator  is  the  soldiers’  environment  (e.g.,  COFT).  In  other 
simulations,  such  as  for  the  battalion  staff  in  a  SIMNET  exercise,  the  simulator  is  removed  from 
the  staff  members  as  they  interact  with  soldiers  who  are  physically  encased  in  the  simulation. 

The  crew  members  in  COFT  are  learning  precise  motor  skills  and  tightly  coordinated  (though 
fairly  simple)  team  skills.  These  skills  can  and  should  become  highly  automated.  On  the  other 
hand,  the  battalion  staff  members  are  learning  a  much  less  routinized,  more  adaptive  kind  of 
activity.  While  both  tank  crew  and  battalion  staffs  are  learning  to  react  quickly  to  environmental 
cues,  the  two  differ  widely  in  the  complexity  of  environmental  cues  and  the  speed  of  needed 
reactions.  Staff  members  are  concerned  with  patterns  of  locations  for  several  dozen  vehicles; 
tank  crews  deal  in  precise  location  of  a  few  vehicle.  Staff  member  reactions  are  in  minutes;  tank 
crews  are  in  tenths  of  seconds.  The  learning  task  is  quite  different.  Kraiger  et  al.  (1997)  argue 
that  training  research  designs  should  become  more  attuned  to  those  differences.  They  have 
suggested  an  elaboration  of  Kirkpatrick’s  (1976)  time  honored  four  levels  of  research  criteria. 
ARI  could  continue  this  line  of  thinking  by  more  explicitly  specifying  the  criteria  appropriate  for 
the  different  types  and  complexity  of  the  skills  being  acquired  and  the  research  methodologies 
needed  to  support  sound  training  decisions.  For  example,  learning  in  the  training  environment,  if 
appropriately  measured,  may  be  an  adequate  index  of  training  effectiveness  for  complex 
cognitive  skills.  ARI  could  provide  much  needed  leadership  is  this  basic  research  area. 

Part-Task  Training 

Research  on  part-task  training  addresses  the  question:  Are  there  ways  in  which  a 
complex  task  can  be  broken  down  into  parts  that  can  be  learned  more  efficiently  than  the  whole 
task  itself?  Part-task  training  using  TADSS  has  been  an  especially  promising  approach  for  flight 
training  and  has  been  studied  in  relation  to  armor  gunnery.  The  approach  enables  repeated 
practice  on  critical  skills  that  do  not  occur  frequently  in  a  whole-task  context  (e.g.,  emergency 
procedures),  allows  trainers  to  focus  on  tasks  where  the  operator  attends  to  other  activities, 
supports  the  training  of  new  response  requirements  for  new  equipment  or  missions,  and  helps 
with  sustainment  of  easily  forgotten  responses  (e.g.,  procedures).  In  addition,  part-task  trainers 
tend  to  be  less  expensive  and  therefore  may  be  distributed  more  widely  than  other  types  of 
training  simulators.  The  advantages  to  the  RC,  with  its  geographic  dispersion  of  troops  and 
equipment,  are  obvious. 

Requirements 

A  part-task  training  procedure  should  specify  a  method  for  partitioning  the  task  for 
training  and  a  method  for  reintegrating  the  task  as  training  progresses.  Three  categories  of 
partitioning  methods  have  been  identified  (Naylor,  1962):  simplification  (changing 
characteristics,  such  as  time  requirements,  to  make  a  task  easier);  fractionation  (dividing  the  task 
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into  separate  time-shared  dimensions,  such  as  procedural  then  perceptual  requirements);  and 
segmentation  (dividing  the  task  on  temporal  or  spatial  dimensions).  Partitioning 
recommendations  in  Army  tactical  training  tend  to  favor  simplification  (e.g.,  crav^^l-walk-run). 

For  example,  training  developed  for  ARNG  battalion  level  battle  staffs  enables  a  sequence  that 
focuses  first  on  information  processing  with  a  few  officers  (C/ST  Staff,  1995),  proceeds  to  staff 
integration  with  a  full  staff  (Hoffinan,  Graves,  Koger,  Flynn,  &  Sever,  1995),  and  culminates 
with  staff  operations  in  a  simulation  of  a  battalion-level  execution  of  a  tactical  mission  (C.H. 
Campbell,  R.C.  Campbell,  Sanders,  Flynn,  &  Myers,  1995).  The  segmentation  strategy  was  used 
to  divide  training  for  the  brigade  staff  decision-making  process  into  a  series  of  training  vignettes 
which  isolate  attention  to  specific  portions  of  the  process  (Graves,  J.  Campbell,  &  Deter,  in 
preparation). 

Three  methods  have  also  been  identified  for  reintegrating  the  task  (Naylor,  1962):  pure 
(separate  practice  on  each  part  before  transfer  to  the  whole  task— A,  B,  C,  ABC);  progressive  (A, 
B,  AB,  C,  ABC);  and  repetitive  (A,  AB,  ABC).  A  frequently  recommended  variation  is  a  reverse 
repetitive  schedule  (C,  BC,  ABC),  sometimes  called  a  backward-chain  schedule.  Wightman  and 
Lintem  (1984)  recommended  segmentation  for  partitioning  and  backward  chaining  for 
reintegrating.  Work  by  Bailey,  Hughes,  and  Jones  (1980)  illustrates  the  recommended  part-task 
procedure.  They  partitioned  a  dive  bomb  task  into  base  leg,  roll-in,  and  final  approach  segments. 
For  integration,  subjects  first  practiced  the  final  approach,  then  practiced  roll-in  and  final 
approach,  and  finally,  practiced  base  leg,  roll-in,  and  final  approach.  Subjects  trained  by  the 
part-task  approach  showed  more  accuracy  and  higher  rates  of  achievement  of  the  criterion  within 
the  time  allotted  than  subjects  trained  by  a  whole-task  method. 

Benefits 

While  the  aviation  psychology  literature  indicates  that  part-task  training  is  effective, 
especially  for  the  segmentation/backward  chaining  technique  (Knerr,  Morrison,  Miunaw,  Stein, 
Sticha,  Hoffinan,  Buede,  &  Holding,  1986),  findings  have  not  universally  endorsed  it.  For 
example,  Adams’  (1960)  review  of  early  research  foimd  that  the  evidence  favored  whole-task 
training  2:1.  Still,  there  has  been  an  increasing  interest  in  part-task  training.  Two  reasons  for 
this  increasing  interest  are  especially  pertinent  for  research  on  part-task  training  in  the  RC 
environment:  applicability  to  transition  and  sustainment  training,  and  cost/time  savings  (Knerr  et 
al.,  1986). 

The  potential  for  part-task  TADSS  is  clearest  for  transition  and  sustainment  training  (as 
opposed  to  initial  training).  The  proliferation  of  operational  equipment  (e.g.,  aircraft)  and  the 
need  to  sustain  skills  through  long  periods  of  peacetime  make  the  application  of  part-task 
TADSS  especially  relevant  for  RC  training.  Effectiveness  for  transition  and  sustainment  varies 
as  a  function  of  task  organization,  complexity,  and  trainee  ability.  Unorganized,  more  complex 
tasks  and  low  trainee  ability  favor  part-task  approaches  (Wells  &  Hagman,  1989). 

The  second  proposed  reason  for  research  on  part-task  TADSS  in  the  RC  is  that  part-task 
training  is  potentially  less  expensive  than  whole-task  training.  Even  without  considering  the  cost 
of  the  TADSS,  time  scores  in  early  research  tended  to  favor  part-task  training.  Even  Adams 
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(1984)  concluded  that  “a  combination  of  part-  and  whole-task  training  may  be  more  cost 
effective  than  whole-task  training  alone”  (p.  64). 

Development  Approaches 

Achieving  those  potential  savings  requires  research  into  ways  to  combine  part-task 
TADSS  training,  whole-task  TADSS  training,  and  operational  equipment  training  in  an  RC 
environment.  Cognitive  task  analysis  techniques  help  identify  skills  to  be  addressed  and 
empirical  approaches  can  help  refine  the  part-task  strategy. 

Cognitive  Task  Analysis.  Cognitive  task  analysis  techniques  are  useful  for  identifying 
skills  that  can  be  trained  with  a  part-task  approach.  For  example,  Schneider’s  (1985)  analysis  of 
a  subtask  of  air  intercept  control  identified  eight  skills  that  could  be  trained  independently  and 
integrated  into  superordinate  subtask  or  task  performance.  Fisk  and  his  colleagues  (Eggemeier, 
Fisk,  Robbins,  &  Lawless,  1988;  Fisk  &  Eggemeier,  1988)  have  developed  a  less  resource 
intensive  front-end  analysis  methodology  that  has  been  used  to  identify  cognitive,  perceptual, 
and  motor  skills  required  for  aircraft  command  and  control  tasks. 

Empirical  Information.  As  an  example  of  an  empirical  approach  to  decide  how  to 
combine  TADSS,  Hart,  Hagman,  and  Bowne  (1990)  measured  the  effects  of  transfer  from  a  part- 
task  trainer  (TopGun)  to  a  whole-task  simulator  (COFT)  and  foimd  the  transfer  to  be  rapid  (one 
20-min  block  significantly  improved  COFT  accuracy). 

Research  Questions 

In  concept,  the  applicability  of  part-task  training  for  the  RC  is  clear.  The  critical  research 
questions  are  specific  to  each  training  requirement  that  part-task  training  might  be  used  to 
address. 

How  should  a  task  be  segmented  for  part-task  training^  A  part-task  training  strategy 
assumes  that  the  part  is  important  and  training  with  a  TADSS  that  represents  it  will  produce 
positive  transfer.  A  substantial  body  of  research  indicates  that  the  validity  of  the  assumption 
depends  on  the  characteristics  of  the  task  and  how  the  task  is  segmented  (Knerr  et  al.,  1986). 
Cognitive  task  and  hierarchical  analysis  approaches  help  partition  the  task  into  meaningful 
components.  Candidate  cognitive  analysis  approaches  are  task  structure  analysis  and  task 
process  analysis.  A  task  structure  analysis  describes  how  knowledge  is  organized  among  the 
people  to  be  trained  (Morrison,  1984).  Task  process  analysis  describes  how  the  activities  that 
make  up  a  task  are  organized.  The  results  of  such  an  analysis  are  expressed  in  condition-action 
rules  (e.g.,  if  the  machine  gim  bolt  is  forward  and  the  safety  is  on  fire,  pull  the  bolt  back). 
Research  should  develop  guidance  on  which  approaches  are  appropriate  for  various  task  types 
(procedural,  perceptual,  perceptual-motor,  and  decisional)  and  proceduralize  the  techniques. 

What  is  the  optimum  mix  of  part-  and  whole-task  devices?  Work  on  identifying 
prerequisite  skills  levels  (often  called  “gates”)  for  whole-task  TADSS  (e.g.,  reticle  aim  levels  to 
change  levels  within  COFT  and  indicate  readiness  for  live-fire)  should  be  extended  to  part-task 
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TADSS  to  indicate  when  training  would  be  more  efficient  on  a  whole-task  TADSS. 

Experimental  designs  should  control  for  trainee  experience  and  ability  level  and  the  criteria  of 
effectiveness  of  candidate  combinations  should  include  time-to-criterion  and  retention. 

TADSS  Fidelity  Requirements 

Research  on  TADSS  fidelity  addresses  the  question:  What  level  of  fidelity  is  required  for 
a  TADSS  to  provide  the  physical  events  and  conditions  that  are  necessary  to  support  training? 
(Hughes,  1979).  The  goal  is  to  provide  procedures  to  determine  what  the  phrase  “necessary  to 
support  training”  means.  The  question  is  important  for  RC  training  because  of  the  impact  that 
development  costs  have  on  the  number  of  TADSS  that  can  be  distributed  to  RC  units.  Like  the 
part-task  trainer,  low-fidelity  trainers  tend  to  be  relatively  inexpensive  and  thus  can  be  fairly 
widely  distributed  on  a  fixed  budget. 

The  proposed  research  considers  two  dimensions  of  fidelity  and  applies  a  range  of 
metrics  to  those  attributes.  The  range  of  attributes  associated  with  each  dimension  and  the 
fidelity  metrics  are  shown  in  Table  1,  which  is  adapted  from  guidance  for  the  development  of 
models  of  aviation  simulators  (Sticha,  Blacksten,  Knerr,  Morrison,  &  Cross,  1986). 

Scope  of  Fidelity 

Table  1  identifies  two  categories  of  design  attributes  for  TADDS,  realism  and 
comprehensiveness.  Eight  more  specific  attributes  are  listed  within  the  two  categories,  with 
examples  of  each  provided.  Table  1  also  proposes  five  criteria  to  inform  decisions  about  whether 
a  TADSS  will  yield  adequate  fidelity.  Training  transfer  is  the  “primary”  criterion,  with  the  other 
four  being  “secondary.”  Obviously,  the  primeiry  criterion  of  transfer  can  only  be  assessed  after 
an  operational  prototype  device  is  built.  The  secondary  criteria,  on  the  other  hand,  can  be 
assessed  during  TADSS  design.  These  secondary  metrics  are  especially  important  given  the 
trend  toward  an  industrial  model  of  acquisition  (COL  R.  Krug,  personal  communication, 
December  12, 1996).  The  industrial  model  maximizes  cost-benefit  ratios  with  short  development 
cycles  and  a  quick  reaction  decision-making  environment  (“fire-aim-fire-aim”).  The  secondary 
metrics  promise  significant  value  for  providing  early  evidence  of  whether  training  devices  are 
effective. 

Physical  measures.  Early  research  on  TADSS  found  that  training  transfer  from  a 
photographic  mock-up  of  a  cockpit  was  comparable  to  training  on  a  high-fidelity  simulator 
(Prophet  &  Boyd,  1970).  In  the  early  1980s,  the  Defense  Science  Board  called  for  greater 
emphasis  on  low-cost  TADSS,  based  in  part  on  the  premise  that  effective  training  can  be 
accomplished  with  TADSS  whose  physical  attributes  differ  substantially  from  the  attributes  of 
the  operational  equipment.  There  are,  of  course,  limits  to  how  far  physical  similarity  can  be 
degraded.  But  it  is  likely  that  dynamic  response  and  sensory  stimuli  can  exhibit  lower  fidelity 
than  operational  equipment  in  a  field  environment  and  still  be  effective.  Research  to  identify 
appropriate  levels  requires  measures  of  parameters  that  can  be  manipulated  to  vary  realism  or 
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comprehensiveness.  For  example,  a  computer-generated  scene  might  be  quantified  at  various 
levels  of  elements  (e.g.,  lines,  polygons). 

Ratings.  Operational  and  developmental  TADSS  are  fi-equently  rated  by  subject  matter 
experts  (SMEs),  for  example,  aviator  ratings  of  the  training  value  of  operational  rotary-wing 
simulators  (McAnulty,  Cross,  &  DeRoush,  1989).  It  has  been  observed  that  soldiers  tend  to  be 
critical  of  low  fidelity  simulations  (J.  Hagman,  personal  communication,  July  23, 1997).  They 
typically  do,  however,  perceive  them  to  be  sufficiently  similar  to  the  operational  equipment.  As 
a  result,  the  applicability  of  ratings  depends  largely  on  methodological  concerns,  especially 
narrowing  the  focus  of  raters  and  training  them  to  offset  biases.  Ratings  are  most  promising 
when  raters  focus  on  specific  attributes,  especially  the  realism  of  dynamic  responses.  Research 
should  assess  training  methods  for  SMEs  (e.g.,  on  sensation  and  perception)  and  develop  scales 
that  minimize  rating  errors  (e.g.,  halo  effects)  for  attribute  ratings. 

In-TADSS  response  measures.  This  metric  compares  an  operator’s  response  in  a 
TADSS  with  responses  in  operational  equipment  under  comparable  circumstances  or  established 
performance  standards.  Sample  metrics  include  peak  performance  level  in  TADSS  compared 
Avith  operational  equipment  or  performance  standards;  type,  frequency,  and  consequences  of 
cognitive  and  motor  errors;  accuracy  of  absolute  judgments  (e.g.,  altitude);  workload  level;  and 
patterns  of  eye  movements  (Sticha  et  al.,  1986).  The  concept  is  an  analog  to  physical  similarity: 
if  training  can  be  effective  despite  departures  fi-om  physical  realism,  can  training  also  be  effective 
despite  differences  in  TADSS  responses  compared  with  operational  equipment  responses? 
Response  measures  are  promising  objective  approaches  to  quantifying  realism  and 
comprehensiveness.  Research  implementing  in-TADSS  response  measures  should  include 
collection  of  data  on  corresponding  responses  in  the  operational  equipment. 

Analytic  measures.  This  class  includes  fidelity  metrics,  other  than  physical  measures,  that 
quantify  the  dynamic  response  and  comprehensiveness  of  TADSS.  In  their  simplest  form,  the 
measures  consist  of  lists  showing  the  range  of  tasks  and  conditions  that  can  be  trained  in  the 
TADSS.  The  approach  has  been  valuable  for  estimating  the  degree  that  TADSS  could  support  tank 
gunnery  conditions  (Hoffinan  &  Morrison,  1988)  and  gunnery  skills  and  knowledges  (Campshure, 
1991).  A  similar  approach  has  been  applied  to  collective  tactical  training  (Burnside,  1990).  These 
applications  are  based  on  a  series  of  yes/no  judgments  concerning  whether  a  condition  or  task 
characteristic  is  supported.  In  some  cases,  it  may  be  desirable  to  add  judgments  of  the  level  of 
realism.  Research  is  needed  to  determine  the  accuracy  of  the  early  judgments  and  to  extend  the 
technique  to  attributes  such  as  dynamic  response  and  sensory  stimuli. 

A  variety  of  models  were  developed  during  the  late  1980s  that  combine  data  about  the 
training  environment,  including  TADSS  fidelity,  to  predict  transfer  to  the  operational 
environment.  The  models  consider  information  such  as  SME  estimates  about  training 
requirements,  task  characteristics,  trainee  population  skills,  TADSS  instructional  features,  and 
TADSS  fidelity  dimensions  (Sticha,  Blacksten,  Buede,  Singer,  Gilligan,  Mumaw,  &  Morrison, 
1990).  Five  of  the  most  promising  models  are: 
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•  OSBATS  (optimization  of  simulation-based  training  systems),  developed  for  the  Army 
oriented  primarily  on  aviation  (Sticha  et  al.,  1990). 

•  FORTE  (forecasting  training  effectiveness),  developed  for  Navy  aviation  TADSS 
(Pfeiffer,  Evans,  &  Ford,  1985). 

•  TECIT  (training  effectiveness  and  cost  iterative  technique),  developed  for  Army  TADSS 
(Goldberg,  1988). 

•  ASTAR  (automated  simulator  test  and  assessment  routine),  developed  to  be  incorporated 
into  the  Department  of  Defense  (DoD)  instructional  system  development  process 
(Companion,  1990). 

•  VALTRAIN  (value  of  training),  which  builds  in  part  on  OSBATS  and  TECIT  (Witmer, 
1991). 

Efforts  to  refine  and  apply  such  models  have  been  largely  dormant  since  1991,  in  part 
because  of  difficulties  in  convincing  military  decision  makers  that  analytic  methods  (e.g.,  the 
secondary  metrics)  produce  valid  results  (Simpson,  1995).  The  comments  cited  earlier  about  the 
need  for  information  to  support  quick  decisions  early  in  the  procurement  of  TADSS  suggest  that, 
ARNG  decision  makers,  at  least,  recognize  the  promise  of  these  methods  to  consolidate  useful 
data  in  situations  that  preclude  experimental  methods. 

Research  Questions 

The  research  questions  related  to  TADSS  fidelity  are  intended  to  revive  research  on 
transfer  prediction  models.  The  scope  of  that  research  includes  selecting  a  model,  improving 
data  to  be  considered,  and  verifying  the  accuracy  of  predictions. 

How  should  secondary  fidelity  criteria  be  combined  to  predict  transfer?  Initial  efforts 
should  provide  information  to  support  selection  of  one  of  the  models  for  individual  training. 
Subsequent  research  should  improve  the  ease  of  applying  the  model  and  provide  case  studies  that 
can  guide  extensions  to  other  TADSS.  Ultimately,  research  should  be  conducted  to  assess  the 
applicability  of  the  refined  model  to  TADSS  that  involve  collective  tasks. 

What  measures  should  be  applied for  each  fidelity  attribute?  Answering  this  question 
entails  compiling  a  complete  inventory  of  measures  needed  to  quantify  attributes  of  realism  and 
comprehensiveness  for  a  given  TADSS.  Inventories  of  measures  based  on  operational  TADSS 
would  be  valuable  for  guiding  analysis  of  TADSS  in  development. 

What  methods  should  be  applied  to  improve  the  quality  of  secondary  fidelity  criteria? 
Avenues  of  research  on  individual  metrics  were  presented  in  the  discussion  of  each  criterion. 
Important  topics  include  development  of  quantifiable  physical  measures,  assessment  of  the 
impact  of  rater  training  on  the  reliability  of  ratings  by  SMEs,  collection  of  data  on  operational 
equipment  responses,  and  development  of  analytic  approaches  for  dynamic  response  and  sensory 
stimuli. 
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How  do  soldiers'  reactions  to  low  fidelity  TADSS  impact  on  the  true  importance  of 
fidelity?  The  negative  reactions  that  soldiers  tend  to  have  toward  low-fidelity  TADSS  raises  an 
interesting  line  of  questioning.  To  what  extent  do  soldiers  attitudes  about  the  TADSS  they  are 
training  with  influence  their  learning?  Can  these  reported  negative  attitudes  have  an  adverse 
effect  on  training  and,  if  so,  do  fidelity  criteria  need  to  include  consideration  of  these  attitudes? 

How  accurate  are  predictions  by  a  transfer  prediction  model?  Validation  data  serve  two 
purposes.  First,  they  enhance  the  acceptability  of  the  model  by  the  Army  leadership.  Second, 
validation  data  help  in  refining  the  model,  especially  simplifying  data  collection  and  interpretation  of 
results.  If  possible,  a  series  of  transfer  studies  as  described  in  the  section  on  cost  trade-offs  should  be 
conducted.  These  studies  should  reflect  systematic  variation  in  fidelity  levels  of  attributes  that  differ 
measurably  fi-om  the  corresponding  attribute  in  the  operational  equipment.  The  research  methods 
should  incorporate  a  validated  instructional  system  (sequence,  tasks,  and  training  support  materials) 
and  control  for  soldier  experience.  For  RC  purposes,  it  would  be  desirable  for  the  experience  level  to 
be  consistent  with  transfer  or  continuation  training.  Training  effectiveness  studies  for  the  close 
contact  technical  trainer  (CCTT)  and  lower-fidelity  SIMNET  would  provide  a  unique  opportunity  to 
study  a  range  of  fidelity  levels  without  having  to  reengineer  a  TADSS  to  vary  attributes. 

As  a  practical  matter,  training  transfer  studies  may  not  be  feasible  because  of  resomce 
constraints  given  the  complexity  of  the  models  (which  may  result  in  more  variations  than  there 
are  subjects,  especially  for  collective  TADSS).  In  that  case,  empirical  data  should  focus  on 
submodels.  For  example,  a  validation  of  the  OSBATS  might  focus  on  predictions  from  the 
Fidelity  Optimization  Module. 

Training  Support  for  TADSS 

Work  with  the  Reserve  Component  Virtual  Training  Program  (RCVTP),  which  has  been 
renamed  the  Virtual  Training  Program  (VTP),  illustrates  one  way  that  research  organizations  add 
value  to  Army  training  developments.  ARI  combined  the  efforts  of  behavioral  scientists,  experts 
in  a  subject  matter  area  (e.g.,  battalion-level  tactical  operations),  and  experts  in  applying  a  given 
TADSS  (e.g.,  SIMNET)  to  develop  instructional  content  for  a  TADSS  and  develop  training 
support  packages  (TSPs)  to  implement  that  instruction. 

Development  of  TADSS-Based  Instruction 

If  a  TADSS  is  to  be  effective,  it  must  be  supported  by  systematic  training  procedures. 
Development  of  those  procedures  includes  extending  the  content  for  existing  TADSS 
applications  and  developing  content  for  new  TADSS  applications. 

Extend  TADSS  instruction.  Implementing  TADSS  has  resulted  in  a  wide  range  of 
instructional  content  that  has  been  shown  to  be  practical  for  delivery  in  an  RC  environment.  As 
more  formal  data  on  effectiveness  accumulate,  it  will  be  desirable  to  develop  additional  instruction 
for  the  various  TADSS.  One  type  of  extension  is  to  convert  instruction  developed  for  one  TADSS 
to  a  different,  similar  TADSS.  The  primary  example  of  such  a  conversion  is  the  development  of 
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scenarios  for  CCTT  based  on  SIMNET  materials  (COL  R.  Krug,  personal  communication, 
December  12, 1996).  A  second  type  of  extension  is  development  of  similar  materials  for  similar 
audiences.  For  example,  the  computer-based  instruction  (CBI)  developed  specifically  for  the 
ARNG  battle  staff  in  a  heavy  battalion  task  force  is  being  extended  to  the  AC  brigade  commander 
and  staff.  The  AC  materials  might  need  to  be  converted  for  RC  use  and  further  materials  might  be 
developed  for  the  staffs  in  other  types  of  RC  battalions.  A  third  type  of  extension  involves 
development  of  additional  modules  for  current  TADSS  applications.  For  example,  additional 
offensive  and  defensive  missions  are  being  developed  for  the  Staff  Group  Trainer  (SGT),  and 
should  be  developed  for  multi-media  interactive  technology  in  the  Armor  Junior  Leader  Tactical 
Training  Program  (D.R.  Jones,  Bullock,  Henriksen,  &  Tkacz,  1991). 

Add  TADSS  instruction.  Interviews  with  RC  officers  identified  two  potential 
applications  of  CBI.  One  officer  requested  the  development  of  a  structure  that  would  enable 
units  to  develop  CBI  on  immediate  needs  (COL  D.  Youngman,  personal  communication, 
December  12, 1996).  The  specific  application  was  CBI  to  implement  revisions  to  the  standing 
operating  procedures.  An  Adjutant  General  (TAG)  requested  a  CBI  program  to  assist  junior 
officers  in  the  transition  from  a  basic  course  or  AC  assignment  to  an  RC  assignment.  The 
instruction  would  prepare  the  officer  to  deal  with  challenges  unique  to  the  RC  environment  (MG 
Kane,  personal  communication,  February  7, 1997). 

Development  of  TSPs 

A  TSP  integrates  training  products,  materials,  and  information  necessary  to  train  one  or 
more  tasks.  For  TADSS  applications,  a  TSP  should  include  a  training  strategy,  task  list,  learning 
objectives,  an  after  action  review  (AAR)  framework  (including  measures  of  performance), 
guidance  to  system  operators,  and  guidance  to  0/Cs.  The  importance  of  such  materials  was 
underscored  in  one  of  our  project  interviews:  “In  Navy  terms,  a  TADSS  [in  this  case  a  Janus 
suite]  without  a  TSP  is  a  very  expensive  anchor”  (COL  R.  Krug,  personal  communication, 
December  12, 1996).  The  research  implication  is  that  development  and  evaluation  of  TSPs 
should  have  a  high  priority  in  projects  to  apply  TADSS.  To  accommodate  RC  time  constraints, 
TSPs  should  be  detailed  enough  to  support  a  turn-key  implementation  that  minimizes  set-up 
requirements  for  RC  trainers.  The  tryout  of  JMSE  materials  in  a  MUTA  environment  (Hoffinan 
et  al.,  1995)  illustrates  the  recommended  approach  to  evaluate  the  full  TADSS  system. 

Research  Questions 

As  a  practical  matter,  the  effectiveness  of  a  TADSS  cannot  be  assessed  unless  the  content 
is  relevant  to  imits’  perceived  needs  and  the  delivery  is  efficient.  Thus  the  questions  discussed 
below  are  directed  toward  assuring  that  relevance  and  efficiency. 

What  content  should  be  developed  for  TADSS  training?  Immediate  candidates  for 
extending  applications  include  conversion  for  similar  TADSS,  applications  for  similar  training 
audiences,  and  adding  modules  to  existing  TADSS  programs.  Candidates  for  new  applications 
include  a  structure  for  locally  produced  CBI  and  a  CBI  program  to  facilitate  the  transition  of 
junior  officers  to  the  RC  environment. 
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How  well  do  TSPs  support  implementation  of  TADSS  training  in  the  RC  environment? 
At  a  minimum,  TSPs  should  be  subjected  to  a  formative  evaluation  process  (including  revision) 
before  a  TADSS  application  is  implemented. 

Computer-Generated  Forces  (CGF) 

Virtual  tactical  engagement  simulation  (TES)  technology  (e.g.,  SIMNET)  allows 
collective  training  in  manned  simulators  projected  onto  a  common  synthetic  battlefield  generated 
by  computer.  While  the  activities  of  the  soldiers  being  trained  determine  the  events,  they  operate 
in  a  context  that  includes  adjacent  units  and  an  opposing  force  (OPFOR).  Virtual  TES  can 
incorporate  role-players  for  other  forces  (e.g.,  the  “World-Class  OPFOR”  in  the  Battle  Command 
Training  Program).  But  using  role-players  increases  support  requirements  and  reduces 
flexibility,  since  a  unit  conducting  the  training  must  either  supply  role  players  fi'om  its  resources 
or  coordinate  with  another  unit  or  agency  to  get  the  role  players.  For  training  economy, 
especially  in  the  RC  environment,  it  is  desirable  for  these  forces  to  be  computer-generated.  The 
research  need  is  to  develop  a  methodology  to  provide  for  CGF  that  act  and  react  realistically  on 
the  simulated  battlefield,  while  maintaining  the  same  concern  for  fidelity  requirements  described 
in  the  previous  section.  Research  on  CGF  should  be  directed  toward  procedures  to  identify  CGF 
requirements  and  to  develop  cost-effective  enhancements  to  realism  in  information  processing 
and  representation. 

CGF  Requirements 

Friendly  CGF  include  adjacent  forces  (e.g.,  flanking,  screening,  supporting,  or  supported 
units),  higher  headquarters  (e.g.,  battalion  and  brigade  for  a  company  exercise),  and  subordinates 
(e.g.,  squad  members  for  military  operations  in  urbanized  terrain).  These  forces  should  operate 
in  a  doctrinally  appropriate  way  given  the  mission,  enemy,  terrain,  fnendly  troops,  and  time 
(METT-T)  conditions.  Performance  requirements  for  fiiendly  CGF  are  well  established  for  most 
combined  arms  units.  The  Blueprint  of  the  Battlefield  (U.S.  Army,  TRADOC,  1992a)  presents  a 
hierarchical  structure  of  functions  (i.e.,  battlefield  operating  systems)  and  subfunctions,  and 
Army  training  and  evaluation  program  mission  training  plans  (ARTEP-MTPs)  describe  tasks 
required  by  functions.  Detailed  analyses  of  task  requirements  tied  to  functions  have  been 
developed  in  analyses  of  battlefield  functions  for  the  heavy  task  force  (Mullen,  1996)  and  of 
command  and  control  for  the  armored  brigade  and  supporting  battalions  (Ford,  Mullen,  & 
Keesling,  1997). 

Identification  of  OPFOR  performance  requirements  is  more  of  a  problem.  Since  the  well- 
defined  doctrine  for  Warsaw  Pact  units  may  not  apply,  METT-T  requirements  will  rely  more  on 
assumptions,  projection,  and  partial  intelligence  than  in  the  past.  If  there  is  to  be  a  “doctrinal” 
OPFOR,  even  controllers  of  the  OPFOR  need  to  be  constrained  from  gamesmanship  that  enable 
them  to  win  rather  than  replicate  OPFOR  practices.  Given  the  uncertainties  associated  with 
defining  a  doctrinal  OPFOR,  reassessing  the  benefits  of  a  standardized  OPFOR  may  be  justified. 
The  alternative— force-on-force  exercises— may  facilitate  training  more  units  in  the  RC  environment. 
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Once  performance  requirements  are  identified,  performance  parameter  values  should  be 
assigned  for  subordinate,  adjacent,  and  OPFOR  elements.  Examples  of  parameter  values  are 
movement  rates  for  dismounted  infantry,  number  of  roimds  for  fighting  vehicles,  defensive 
fi-ontages  for  units,  and  kill  probabilities.  Values  might  be  expressed  as  discrete  values,  ranges, 
or  rules  (for  non-numerical  characteristics). 

CGF  Enhancements 

For  CGF  to  be  effective,  capabilities  must  be  enhanced  in  two  areas:  information 
processing  and  representation  of  dismounted  elements.  CGF  are  currently  limited  to  avoiding 
fixed  obstacles  and  choosing  among  predefined  alternatives.  Developments  to  enhance 
information  processing  should  include  improved  terrain  reasoning,  movement  control,  situation 
assessment,  tactics,  and  cooperation  (Reece,  1994). 

As  simulations  such  as  CCTT  incorporate  dismounted  soldiers  (e.g.,  infantry,  engineer), 
the  cost  of  representing  the  dismounts  will  become  a  major  factor.  For  example,  a  combat 
vehicle  can  be  represented  realistically  with  200  to  300  polygons.  Similar  realism  for  one  human 
figure  requires  1600  polygons  (Sticha,  R.C.  Campbell,  &  Schwalm,  1996).  The  costs  of  realistic 
representation  will  certainly  decrease,  for  example  through  innovations  in  JACK,  the  3- 
dimensional  human  figure  model  used  in  computer-aided  design.  The  RC  concern  is  to  be  sure 
that  the  added  realism  (which  restricts  distribution  of  TADSS)  is  justified  in  terms  of  motivation 
and  training  transfer. 

Research  Questions 

Are  computer-generated  OPFOR  justified?  One  of  the  assumptions  for  CGF  is  that 
tactical  training  requires  a  doctrinal  OPFOR.  With  the  proliferation  of  potential  scenarios,  that 
assumption  may  not  apply  (COL  Gordon,  personal  communication,  February  7, 1997).  The 
question  is  important  for  RC  training  because  force-on-force  exercises  where  both  sides  apply 
U.S.  doctrine  would  increase  the  number  of  units  who  could  be  trained  in  a  given  time  (e.g.,  one 
platoon  defending  against  one  or  two  attacking  platoons). 

How  should  requirements  and  parameters  for  CGF  be  established?  Establishment  of 
requirements  and  parameters  involves  interaction  between  training  developers  and  SMEs.  The 
focus  of  research  should  develop  a  methodology  for  identification  of  requirements  and 
proceduralize  the  assignment  of  parameter  values. 

How  are  performance  and  training  effectiveness  affected  by  simplifications  in  the 
representation  of  CGF?  This  question  is  similar  to  questions  on  realism  for  a  TADSS.  In  this 
case,  research  would  systematically  vary  levels  of  CGF  realism  and  document  the  effect  on  the 
cost  and  transfer  effectiveness  of  the  TADSS.  Although  development  of  CGF  is  not  anticipated 
to  be  an  ARI  activity,  the  findings  should  be  suitable  for  guiding  refinement  of  current  TADSS 
and  developing  emerging  TADSS. 
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Micro-Level  Strategies  for  TADSS 


Macro-level  strategies  (discussed  under  Training  Management)  provide  overall  goals  for 
training  development  and  planning.  For  those  goals  to  be  achieved,  each  TADSS  must  be 
addressed  by  a  micro-level  strategy  for  training  execution.  Components  of  such  a  strategy 
include  identification  and  organi2ation  of  training  objectives,  assignment  of  TADSS  to 
objectives,  allocation  of  training  times  to  objective/device  combinations,  the  specification  of  the 
content  of  TADSS  training,  and  the  overall  management  of  TADSS  training  (Morrison, 
Campshure,  &  Doyle,  1991).  The  overall  management  component  of  the  strategy  extends  the 
gates  concept  from  the  macro-level  strategy  to  determine  when  an  individual  or  unit  has  reached 
a  level  of  proficiency  to  justify  moving  to  a  different  TADSS,  field  training,  or  a  higher  echelon. 
The  COFT-based  gunnery  training  strategy  (Hagman  &  Smith,  1996)  illustrates  how  a  detailed 
micro-level  strategy  incorporating  a  specific  TADSS  can  reduce  training  time. 

Development  and  validation  of  each  micro-level  strategy  offers  an  opportunity  to  assess 
the  effects  of  learning  variables  on  skill  acquisition,  retention,  and  transfer.  While  the  emphasis 
is  on  maximizing  the  effectiveness  and  efficiency  of  a  TADSS,  researchers  should  document  the 
impact  of  various  approaches  related  to  the  conduct  of  AARs,  timing  and  distribution  of 
feedback,  and  implementation  of  methods  to  enhance  retention. 

AAR.  The  AAR  is  the  primary  form  of  performance  feedback  for  tactical  and  battle  staff 
training.  Micro-level  strategies  for  the  RC  should  identify  techniques  that  blend  effectiveness 
and  efficiency.  One  recommendation  for  enhancing  the  quality  of  the  feedback  is  to  compare 
staff  and  unit  responses  to  expert  responses,  sometimes  termed  “A  Way”  (Brown,  1994).  The 
impact  of  such  an  approach,  while  promising,  has  not  been  demonstrated.  Expert  comparisons 
might  well  be  compared  to  improvement-based  feedback,  which  has  been  well  received  by 
ARNG  units  in  the  VTP  (Hoffman  et  al.,  1995).  Another  distinction  in  AAR  techniques 
concerns  directed  versus  discovery  learning.  In  the  recommended  discovery  technique,  0/Cs 
lead  discussions  so  that  imit  members  rather  than  the  0/C  identify  unit  strengths  and  weaknesses. 
It  is  possible,  however,  that  discovery  feedback  techniques  are  less  effective  for  ARNG  units 
than  for  AC  units,  especieilly  in  time-constrained  IDT. 

Timing  and  distribution  of  feedback.  The  variation  here  concerns  whether  augmented 
feedback  should  be  presented  during  training  in  the  form  of  “extra”  cues  or  deferred  to  a  natural 
break  in  training.  Augmented  feedback  (e.g.,  a  buzzer  that  sounds  when  a  guimer’s  sight  picture 
is  correct)  tends  to  speed  up  learning  but  does  not  transfer  to  a  nonaugmented  condition  (Wells  & 
Hagman,  1989).  Adaptive  withdrawal  (providing  augmented  feedback  only  when  responses 
deviate  significantly  off  course)  may  increase  the  efficiency  of  TADSS  training. 

Enhancements  to  long-term  retention.  Principles  to  enhance  long-term  retention  have 
been  established  for  many  years.  These  principles  include  the  following;  (a)  forgetting  increases 
as  a  positive  function  of  the  retention  interval,  (b)  over-learning  is  beneficial  for  retention,  (c) 
relearning  after  a  retention  interval  is  more  rapid  than  the  original  learning,  and  (d)  discrete 
procedural  responses  are  forgotten  more  readily  than  continuous  motor  skills  (Adams,  1987). 
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Studies  that  illustrate  the  application  of  such  principles  (and  confirm  their  effectiveness)  would 
help  in  the  development  of  future  micro-level  strategies. 

Research  Questions 

How  should  TADSS-based  training  be  implemented  in  the  RC  environment?  This 
question  considers  the  adequacy  of  components  of  a  micro-level  strategy  for  implementing  a 
training  approach.  Since  the  strategy  is  part  of  the  TSP,  it  should  be  evaluated  in  terms  of  clarity 
and  acceptability  to  trainers. 

What  learning  variables  promote  skill  acquisition,  retention,  and  transfer?  Illustrative 
variables  include  AAR  techniques,  enhanced  feedback,  and  application  of  established  principles  for 
enhancing  long-term  retention.  The  intent  is  to  establish  a  fiumework  for  systematically  varying 
characteristics  when  strategies  are  implemented  so  that  effective  principles  can  be  identified. 

Performance  Measurement 

The  importance  of  performance  measures  has  been  well  established  for  individual 
training.  The  emphasis  in  this  area  is  on  extending  performance  measurement  to  collective 
training  by  identifying  quantifiable  indices  of  collective  performance.  Two  factors  make 
collective  performance  measurement  difficult.  The  first  difficulty  is  the  complex  and 
heterogeneous  nature  of  performance-multiple  independent  actions  by  units  affect  multiple 
outcomes,  making  it  difficult  to  summarize  performance  in  meaningful  and  understandable  ways. 
The  second  difficulty  is  that  collective  performance  measures  at  battalion  and  higher  echelons 
may  be  viewed  as  impinging  on  the  commander’s  prerogative  to  assess  the  unit.  Despite  the 
difficulties,  collective  performance  measures  are  vital  for  diagnostic  pmposes  and  for  proficiency 
assessment. 

Diagnostic  Measures 

Diagnostic  measures  are  part  of  the  micro-strategy  for  a  TADSS  and  also  apply  to  field 
training  such  as  Lanes  (Ashworth  et  al.,  1992).  The  Lanes  concept  calls  for  providing  training 
support  from  outside  the  unit  (e.g.,  AC  0/Cs,  pre-packaged  training  scenarios).  In  considering 
the  need  for  performance  measurement,  it  is  important  to  remember  that  TADSS  and  field 
exercises  do  not  in  and  of  themselves  provide  instruction.  Unless  trainers  intervene,  typically 
through  AARs,  the  exercises  only  provide  practice.  Thus  the  skill  and  expertise  of  0/Cs  are  vital 
components  of  effective  collective  training.  Diagnostic  measures  enhance  that  expertise  by 
providing  a  functional  structure  for  AARs  and  objective  baselines  for  analyzing  performance. 

Functional  Structure 

A  variety  of  functional  structures  have  long  been  familiar  to  Army  collective  training 
(e.g.,  move,  shoot,  and  communicate).  Two  of  the  most  pervasive  recent  structmes  are  ARTEP- 
MTPs  and  Battlefield  Operating  Systems  (BOSs).  The  ARTEP-MTPs  list  tasks  by  mission  and 
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provide  standards  of  performance  and  performance  details.  The  BOSs  are  seven  fimctional  areas, 
adopted  by  the  National  Training  Center  (NTC),  that  encompass  tactical  operations.  ARI  has 
further  elaborated  the  BOS  structure  through  a  series  of  projects  to  analyze  battlefield  functions 
(BFs),  formerly  critical  combat  functions  (CCFs),  illustrated  for  a  subset  of  BOS  in  Table  2 
(Ford  et  al.,  1997). 


Table  2 


Sample  Battlefield  Fimctions  Grouped  by  Three  BOS 


BOS 

Battlefield  Function  (BF) 

Applies  to: 
Bn  Bde 

Intelligence 

1. 

Conduct  Intelligence  Planning 

X 

X 

2. 

Collect  Information 

X 

X 

3. 

Process  Information 

X 

X 

4. 

Disseminate  Intelligence 

X 

X 

Maneuver 

5. 

Conduct  Tactical  Movement 

X 

X 

6. 

Engage  Enemy  with  Direct  Fire  and  Maneuver 

X 

Fire  Support 

7. 

Employ  Mortars 

X 

8. 

Employ  Field  Artillery 

X 

X 

9. 

Employ  Close  Air  Support 

X 

X 

10. 

Conduct  Electronic  Collection  and  Jamming 

11. 

Conduct  Battlefield  PSYOP 

12. 

Employ  Chemical  Weapons 

13. 

Conduct  Counter  Target  Acquisition  Operations 

14. 

Employ  Naval  Surface  Fires 

X 

15. 

Coordinate,  Synchronize,  and  Integrate  Fire  Support 

X 

X 

Each  structure  has  advantages  and  disadvantages.  The  ARTEP-MTP  structure  applies 
well  at  the  platoon  level  and  below,  but  the  scope  of  task  detailing  may  be  too  large  at  higher 
echelons.  Given  highly  skilled  0/Cs,  the  BOS  structure  works  well  at  higher  echelons,  but  most 
of  the  BOSs  are  too  broad  to  guide  remediation,  while  orienting  on  individual  BOSs  tends  to 
obscure  synchronization  among  operating  systems.  The  BFs  have  finer  detail  and  explicitly 
address  synchronization  across  systems  and  units,  but  the  level  of  detail  makes  paper-based 
application  difficult  for  most  0/Cs  and  the  analyses  do  not  have  doctrinal  status. 

The  research  need  is  to  identify  the  structure  and  level  of  guidance  0/Cs  need  to  diagnose 
unit  performance  by  unit  type  and  echelon.  Products  should  include  prototype  observer  guides 
based  on  the  recommended  structures. 
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Objective  Baselines 

One  of  the  benefits  of  TADS  S  is  the  capability  to  collect  large  quantities  of  historical  data 
concerning  combined  arms  training  events.  For  example,  the  Unit  Performance  Assessment 
System  (UPAS)  and  the  modified  semi-automated  forces  (ModSAF)  database  in  SIMNET 
automate  the  collection,  integration,  and  display  of  performance  data.  The  performance  data 
have  been  tied  to  ARTEP-MTP  tasks  (Meliza,  Bessemer,  Burnside,  &  Shlechter,  1992).  The 
immediate  application  of  UPAS  and  ModSAF  is  to  enhance  the  AAR  by  playing  back  the 
mission.  The  feedback  function  could  be  enhanced  further  through  the  development  of  baseline 
information  to  indicate  how  unit  performance  compares  not  only  with  previous  performance  by 
the  unit,  but  also  with  other  similar  units.  Such  information  would  be  especially  useful  for 
indicating  when  units  have  achieved  gate  qualifications  for  moving  to  more  demanding  events. 

Research  Questions 

What  level  of  detail  is  needed for  diagnostic  instruments?  This  question  balances  the 
reliability  of  diagnostic  assessments  with  acceptability  of  the  instruments.  The  reliability  issue 
is:  Do  two  or  more  qualified  trainers  who  observe  the  same  behavior  agree  on  the  unit’s  strengths 
and  weaknesses?  At  the  same  time,  researchers  should  determine  whether  0/Cs  who  conduct  the 
TADSS-based  or  field-based  training  have  sufficient  flexibility  to  exercise  their  subject-matter 
expertise.  The  question  might  well  be  addressed  in  the  context  of  0/C  certification  for  TADSS- 
based  training. 

Proficiency  Assessment 

Assessing  training  effectiveness  requires  measures  of  performance  both  during  training 
and  on  the  criterion.  Most  designs  for  assessing  TADSS  effectiveness  use  TADSS-based 
predictor  measures  and  field  exercises  as  the  criterion  behavior.  While  variations  are  possible, 
that  distinction  is  useful  for  discussing  research  requirements.  Regardless  of  the  medium, 
measures  must  be  valid,  targeting  the  processes  and  outcomes  associated  with  the  performance 
domain,  and  reliable,  resulting  in  consistent  results  over  replications  of  performance. 

TADSS-Based  Assessment 

TADSS-based  measures  can  be  highly  appropriate  for  training  effectiveness  analysis. 
These  measmes  have  fewer  factors  to  threaten  validity  (compared  to  measures  in  an  operational 
environment)  and  can  be  manipulated  (e.g.,  lengthened)  to  enhance  reliability.  For  example, 
Graham  (1986)  found  that  a  COFT-based  test  of  gunner  performance  exhibited  high  reliability 
and  enabled  more  detailed  analysis  of  gunnery  performance  than  would  have  been  possible  in  a 
field  environment.  Hagman  and  Smith  (1996)  developed  a  similar  test  on  a  later  COFT  version 
and  found  that  the  COFT  test  was  a  good  predictor  of  live-fire  performance.  While  the 
relationship  between  COFT  and  live-fire  had  long  been  assumed,  that  study  was  the  strongest 
confirmation,  in  part  because  the  sample  of  ARNG  crews  included  a  range  of  gunnery  ability. 
Since  a  range  in  both  predictor  and  criterion  performance  is  needed  to  study  transfer  and 
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prediction,  including  ARNG  units  in  the  sample  may  help  assure  the  needed  variation.  Hagman 
(in  press)  extended  the  analytic  approach  from  the  COFT  prediction  work  to  a  rifle 
marksmanship  TADSS  (the  EST)  and  again  found  a  high  level  of  prediction. 

The  value  of  TADSS-based  assessments  extends  beyond  confirming  the  effectiveness  of 
the  TADSS.  The  assessments  also  enable  efficiencies  in  unit  training  by  (a)  identifying  crews 
and  individuals  who  are  ready  for  live  fire  (Hagman,  in  press;  Hagman  &  Morrison,  1996),  (b) 
supporting  development  of  combat  attrition  models  (Graham,  1986),  and  (c)  helping  in  the 
selection  of  ARNG  crewmen  (COL  R.  Krug,  personal  communication,  December  12, 1996). 
Research  support  of  TADSS-based  assessment  should  include  documenting  and  improving 
psychometric  properties  of  the  measures  and  establishing  standards  that  indicate  gates  for 
moving  to  more  difficult  exercises  or  to  an  operational  training  environment  (e.g.,  FTX,  live 
fire). 


Operational  Measures  of  Effectiveness 

Studies  of  training  effectiveness  ultimately  require  measiues  of  effectiveness  (MOE)  on 
operational  equipment  in  a  realistic  environment.  Because  of  the  desire  for  high  content  validity, 
researchers  will  often  look  to  highly  realistic  training  environments,  especially  instrumented 
combat  training  centers  (CTCs),  for  MOE  for  training  effectiveness  analyses.  The  problem 
comes  when  researchers  try  to  use  0/C  comments  and  instrumented  data  as  criterion  measures. 

It  is  rare  when  measures  designed  for  diagnosis  are  also  appropriate  to  be  MOE. 

In  some  cases,  “normally”  generated  CTC  data  have  been  suitable  for  criterion  piuposes. 
For  example,  casualty  exchange  data  from  several  years  at  the  NTC  showed  a  significant 
relationship  to  operating  tempo  (OPTEMPO)  estimates  (Hiller,  McFann,  &  Lehowicz,  1990).  In 
most  cases,  however,  the  experience  reported  by  Rand  researchers  applies:  “For  every. .  .study 
we  have  done  at  the  NTC,  we  have  had  to  develop  data  items  and  collection  methods  specific  to 
the  study”  (Hodges,  1994,  p.  451).  As  an  example,  researchers  in  ARI’s  Project  Determinants 
developed  separate  rating  forms  targeted  to  a  given  echelon  (e.g.,  platoon)  which  the  0/Cs  agreed 
to  complete  (Keesling,  Ford,  &  Harrison,  1994).  As  another  example.  Institute  for  Defense 
Analyses  (IDA)  scientists  and  SMEs  directly  observed  ARNG  units  at  NTC  and  rated  their 
performance  during  rotations  in  1996  (Dr.  Morrison,  personal  communication,  February  7, 

1997). 


A  review  of  cost-effectiveness  analysis  across  Services  (Simpson,  1995)  found 
development  of  MOE  for  collective  training  to  be  a  fragmented  area  that  “remains  an  art  rather 
than  a  science”  (p.  18).  Simpson  offered  eight  generalizations  to  guide  the  art:  (a)  tasks  tend  to 
exist  in  structures  (e.g.,  hierarchies,  functional  categories);  (b)  the  structure  requires  definition; 

(c)  tasks  at  each  level  require  definition;  (d)  interactions  among  levels  require  definition;  (e) 
tasks  should  be  selected  which  focus  on  the  interests  of  the  training-effectiveness  analysis;  (f) 
tasks  are  converted  to  MOE;  (g)  product  MOE  (mission  accomplishment)  are  the  most  important; 
and  (h)  process  MOE  assess  the  internal  workings  of  the  collective  unit. 
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As  described  earlier,  most  structures  for  Army  collective  training  are  based  on  operational 
fimctions.  The  structures  might  be  more  suitable  for  assessment  if  they  are  augmented  by 
cognitive  analyses  that  define  the  interactions  of  knowledge,  cognitive  processes,  and  behavior  in 
determining  performance  (J.  P.  Campbell,  1991).  For  example,  assessment  of  team  processes 
might  be  structured  into  components  such  as  leadership,  communication,  adaptability,  and 
coordination  and  cooperation  (Swezey  &,  Salas,  1992). 


ARI’s  work  related  to  BFs  has  included  development  of  assessment  packages  that  are 
consistent  with  Simpson’s  generalizations:  the  structure  of  tasks  is  defined  fimctionally,  tasks  at 
each  level  are  defined,  interactions  between  levels  and  across  operating  systems  are  defined,  and 
product  and  process  MOE  are  presented  (Ford  et  al.,  1997).  The  assessment  packages  are 
structured  to  support  assessment  at  increasing  levels  of  specificity: 


BF: 

Outcome: 

Component: 

Task: 


Direct  and  lead  unit  during  preparation  phase  of  battle  (BF  19). 
Subordinate  leaders  demonstrate  an  understanding  of  the  mission, 
commander’s  intent,  and  mission  essential  tasks. 

Briefback  or  rehearsal. 

Brigade  S3  ensures  brigade  combat  power  is  synchronized  in  terms 
of  timing  to  support  brigade  movement  and  maneuver. 


The  assessment  packages  have  been  implemented  on  only  a  partial  basis  to  assess  field  training 
(e.g.,  IDA  used  task  definitions  to  assess  process  of  ARNG  battalions  at  NTC). 


Most  assessments  of  collective  tactical  training  include  process  MOE,  such  as  checklists 
based  on  ARTEP-MTP  task  definitions.  Such  paper-based  tools  can  be  imposing  and  cumbersome. 
Two  lines  of  research  can  facilitate  process  measurement.  The  first  line  of  research  is  to  provide 
information  to  focus  assessment  on  critical  requirements.  To  some  extent,  identification  of  tasks  to 
be  observed  depends  on  the  unit’s  level  of  qualification  and  requirements  of  the  METT-T. 
Development  of  MOE  for  individual  tasks  is  frequently  guided  by  assessments  of  task  criticality  for 
a  variety  of  METT-T  conditions.  Similar  data  on  collective  tasks  would  be  useful. 


The  second  line  of  research  is  to  develop  computer-based  tools  to  support  task-based 
assessments.  An  example  of  such  a  tool  is  the  Electronic  Collection  Instrument  (ECI),  sometimes 
called  the  electronic  clipboard.  ARJ  demonstrated  the  potential  of  the  ECI  during  Army  Warfighter 
Experiment  Desert  Hammer  VI  in  1994.  That  work  demonstrated  the  capability  to  use  ECI  to 
record  data  in  the  field  and  upload  it  to  a  central  database  via  radio.  Despite  that  proof  of  principle, 
the  ECI  was  not  accepted  by  0/Cs.  When  given  an  option  of  collecting  follow-up  data  using  ECI 
or  a  paper-based  format,  almost  all  0/Cs  chose  paper.  Advances  in  technology  will  certainly 
enhance  the  practicality  of  tools  like  ECI  (e.g.,  improving  readability  of  the  monitor  in  bright  light), 
but  if  computer-based  tools  are  to  be  applied  for  training  effectiveness  assessments,  researchers 
must  still  consult  with  0/Cs  during  the  design  of  the  application. 
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Research  Questions 

How  should  performance  assessment  measurement  systems  be  structured?  Two 
research  angles  are  involved  in  addressing  this  question.  The  first  looks  at  the  effect  of 
augmenting  functional  analyses  with  cognitive  analyses  and  empirical  data  on  criticality.  The 
second  considers  how  a  measurement  system  might  be  implemented,  including  product  and 
process  MOE.  The  acceptability  of  the  full  system  should  be  considered,  with  the  crucial 
opinions  coming  from  unit  commanders  and  TRADOC  proponents. 

What  is  the  reliability  and  validity  of  MOE?  The  purpose  of  assessing  the  psychometric 
properties  of  MOE  is  to  identify  and,  ultimately,  control  measurement  errors  that  may  be 
introduced  from  the  context  of  performance  and  from  methods  of  measuring.  While  reliability  and 
validity  are  concerns  in  TADSS-based  and  operational  MOE,  the  TADSS-based  MOE  are  more 
appropriate  media  for  studying  reliability  and  validity  issues.  The  benefits  of  TADSS-based  MOE 
result  from  the  possibility  of  including  multiple  replications  of  unit  performance,  enabling  the 
application  of  structural  equation  techniques  (e.g.,  LISREL)  and  generalizability  theory. 

Training  Management 

The  Standard  Army  Training  System  (SATS)  is  an  automated  training  management 
system  that  is  a  major  component  of  the  Army  Training  XXI  campaign  to  produce  a  training 
system  for  an  information-age  Army  (Dugan,  1997).  SATS  will  support  training  development 
and  scheduling  by  assisting  commanders  and  other  unit  trainers  in  the  production  of  training 
schedules,  calendars,  plans,  resource  requirements,  evaluations,  assessments,  and  readiness 
reports.  ARI  can  support  the  implementation  of  SATS  in  the  RC  by  developing  macro  strategies 
for  ARNG  units  and  developing  and  maintaining  a  performance-oriented  database. 

Macro-Level  Training  Strategies 

A  training  strategy  is  the  systematic  configuration  of  instructional  content  and  methods 
intended  to  enhance  training  effectiveness  (Morrison  &  Hagman,  1994).  The  highest  level  is  a 
macro-level  strategy  that  lays  out  a  unit’s  long-term  training  program  (e.g.,  armor  battalion  2- 
year  strategy).  The  basic  macro-level  strategy  for  Army  unit  training  is  established  by  the 
Combined  Arms  Training  Strategy  (CATS)  (U.S.  Army,  TRADOC,  1992b).  CATS  presents 
events  for  various  battalion  types  and  explains  how  the  event-based  strategy  interfaces  with  the 
related  process  of  training  development,  budget  management,  requirements  determination,  and 
materiel  acquisition.  The  resourcing  model  for  training,  the  Battalion-Level  Training  Model 
(BLTM),  is  based  on  CATS.  In  order  to  revise  BLTM  to  reflect  savings  from  existing  TADSS 
(e.g.,  Janus,  SIMNET)  and  emerging  TADSS  (especially  CCTT),  ARI  was  asked  to  develop  a 
new  CATS.  The  resulting  task-based  CATS  provides  more  complete  descriptions  of  training; 
identifying  tasks  to  be  trained  by  each  event  at  each  echelon,  specifying  the  frequency  with 
which  tasks  must  be  trained  to  sustain  readiness,  and  laying  out  a  menu  of  events  and  media  to 
train  the  tasks  (Keenan  et  al.,  1996).  The  macro-level  strategy  for  a  particular  echelon  of  a 
particular  imit  (e.g.,  a  battery  of  a  field  artillery  battalion)  is  laid  out  as  a  matrix  in  which  the 
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tasks  to  be  trained,  identified  in  the  first  column,  define  rows  that  are  completed  with  entries  for 
each  of  the  columns  laid  out  in  Figure  1.  Matrices  were  developed  for  each  echelon  (battalion, 
staff,  company,  platoon,  squad/crew)  for  1 1  battalion  types.  The  matrices  and  supporting  2-year 
calendars  were  validated  with  proponent  schools,  battalion  commanders  in  Forces  Command 
(FORSCOM),  and  battalion  commanders  in  U.S.  Army  Europe  (USAREUR). 


Task 

FreqV 

Means 

Estimated 

Quality 

Training  Unit 

Prerequisite 

Interval 

Duration 

(Aja 

lAudiencel 

Remarks 

(Includes  purpose  of 
event;  outcome  being 
supported;  comments 
about  execution  of  the 
event;  constraints  posed 
by  TADSS;  etc.) 


Figure  1.  CATS  model  data  elements. 

Three  of  the  conclusions  from  CATS  development  suggest  that  those  strategies  might 
profitably  be  adapted  for  RC  training.  First,  the  CATS  matrices  were  found  to  facilitate  the 
development,  communication,  coordination,  and  resourcing  of  task-based  training  strategies. 
Second,  the  CATS  showed  potential  savings  fi'om  integrating  TADSS  (e.g.,  8%  reduction  in  tank 
mileage  for  a  T-1  armor  battalion  over  a  2-year  cycle).  Third,  leaders  and  trainers  considered  the 
prototype  matrices  to  be  accurate,  complete,  and  useful  (i.e.,  “road  map  to  planning  and 
execution  of  battalion  training”)  (Keenan  et  al.,  1996). 

Commanders  and  other  unit  trainers  will  be  able  to  access  CATS  for  the  1 1  battalion 
types  with  the  introduction  of  S ATS  4. 1 .  The  CATS  analyses  is  currently  being  extended  for  AC 
brigades. 

RC  training  would  also  benefit  fi-om  a  macro-level  strategy  that  clarifies  how  TADSS 
should  be  integrated  into  unit  training.  The  strategy  might  be  an  especially  useful  tool  for 
implementing  a  lower-echelon  emphasis  such  as  BOLD  SHIFT  (Ashworth  et  al.,  1992).  The 
most  efficient  way  to  develop  such  a  strategy  would  be  to  adapt  the  battalion  CATS  matrices  and 
calendars  for  RC  units.  One  requirement  for  the  adaptation  is  to  determine  the  cycle  and 
culminating  events  to  be  addressed.  The  AC  analyses,  which  focused  on  T-1  imits,  used  a  2-year 
cycle  tied  to  a  CTC  rotation  for  all  units.  Greater  variation  may  be  required  for  RC  battalions. 

After  the  matrices  are  adapted,  researchers  should  monitor  their  application  in  a  sample  of 
units.  One  of  the  concerns  during  the  monitoring  will  be  to  document  the  impact  of  basis  of 
issue  of  a  TADSS  on  the  implementation  of  TADSS  training.  This  is  a  vital  concern  for 
projecting  cost  savings  in  the  RC.  For  example,  work  with  battalions  in  Europe  found  that  units 
located  more  than  2  hr  from  a  SIMNET  site  rarely  incorporated  that  TADSS  (Keenan  et  al., 

1996).  At  a  minimum,  extensive  travel  to  a  TADSS  will  reduce  the  number  of  tasks  or 
repetitions  in  the  training. 
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Research  Questions 

What  tasks  should  be  trained  in  TADSS-based  and field  events  during  an  RC  unites 
training  cycle?  Answering  this  question  involves  systematic  review  with  proponent  agencies 
and  experienced  commanders  to  convert  CATS  matrices  and  calendars  that  have  been  developed 
for  the  AC  to  RC  battalion  types.  The  conversion  includes  defining  a  meaningful  training  cycle 
and  culminating  event  for  each  battalion  type.  For  example,  an  enhanced  armor  battalion  might 
have  a  1-year  cycle  culminating  in  a  platoon  battlerun.  Each  macro-level  strategy  should  be 
implemented  in  at  least  one  RC  battalion  to  confirm  its  practicality. 

How  does  the  basis  of  issue  of  TADSS  affect  the  macro-level  strategy  for  RC  units? 

The  immediate  purpose  of  this  question  is  to  guide  distribution  of  TADSS.  In  addition, 
answering  the  question  will  help  identify  alternatives  to  the  macro-level  strategy  in  the  event  that 
projected  TADSS  are  not  available. 

Database  Development  and  Maintenance 

ARI  has  developed  a  prototype  database  for  ARNG  armor  and  mechanized  infantry  units 
that  illustrates  the  potential  for  extending  current  SATS  components  to  include  information  on 
resources  and  performance.  The  ARI  database  is  designed  to  support  development  of  gunnery 
training  strategies  by  documenting  TADSS  implementation  and  training  usage,  measurement  of 
TADSS  performance,  and  relevant  criterion  measures  (R.E.  Jones,  1996).  The  scope  of  data  by 
level  is  illustrated  in  Table  3  (Keesling  8c  Clifton,  1994).  The  database  was  populated  with  data 
for  eight  ARNG  brigades  for  FY94. 

Table  3 


Scope  of  Data  in  Training  Management  Database 


Level 

Sample  Data 

Individual 

Time  in  service 

Armed  Forces  Qualification  Test 

Individual  weapons  qualification 

Crew/Squad 

Gunnery  table  scores  (preferably  first  run) 

COFT  attainment  level 

Company 

Fuel  consumption 

Training  Assessment  Model  (TAM)  ratings 

Battalion 

TAM  ratings 

Ammunition  usage 

Keesling  and  Clifton  (1994)  identified  how  such  data  can  be  analyzed  to  support  training 
management.  The  description  of  prototype  analytic  approaches  was  described  in  terms  of  the 
adaptation  of  a  model  of  applied  research  developed  by  Wold  (1970)  shown  in  Table  4.  The 
general  categories  of  approaches  include  nonexperimental  and  experimental  description, 
nonexperimental  explanation,  and  experimental  explanation. 


29 


Table  4 


Ends  and  Means  in  Applied  Statistics _ 

Ends  Means 


Nonexperimental 

Experimental 

Description 

Descriptive  statistics 

Confidence  intervals 

Longitudinal  descriptions 

Same  means,  with  focus  on 
differences  among  experimental 
conditions 

Explanation 

Causal  inference  from  quasi¬ 
experiments 

Time-series 

Multiple  time-series 

Causal  models  for  nonexperimental 
data 

Comparison  of  treatment  conditions 

Factorial  experiments 

Analysis  of  variance 

Nonexperimental  and  Experimental  Description 

Description  analyses  use  arithmetic  and  graphical  techniques  to  control  random 
fluctuations.  The  difference  between  the  nonexperimental  and  experimental  means  depends  on 
whether  units  can  be  assigned  randomly  to  experimental  conditions.  The  techniques  help  assess 
effects  of  moderating  variables  (e.g.,  crew  turbulence)  and  monitor  trends  in  performance 
indicators  (e.g.,  gunnery  table  scores  and  COFT  reticle  aim  level).  Such  descriptions  can 
improve  training  management  by  providing  feedback  so  that  commanders  can  compare  their 
current  training  program  with  programs  from  previous  years  and  determine  whether  quality  of 
performance  changes. 

Nonexperimental  Explanation 

The  first  set  of  nonexperimental  explanations-time-series  methods-are  a  variation  of  trend 
line  analysis.  They  come  into  play  when  a  change  in  policy  or  practice  is  introduced.  Keesling  and 
Clifton  (1994)  suggested  the  example  of  the  introduction  of  the  Guard  Unit  Armory  Device  Full- 
Crew  Interactive  Simulation  Trainer— Armor  (GUARDFIST  I).  If  Tank  Table  VIII  (gunnery 
qualification)  scores  over  several  years  were  available  for  a  battalion,  the  time-series  design  would 
monitor  whether  the  anticipated  improvement  in  Tank  Table  VIII  scores  occurred.  If  GUARDFIST 
I  were  introduced  to  some  battalions  before  others,  a  multiple  time-series  design  would  add  a 
comparison  of  performance  trend  lines  for  units  with  the  TADSS  to  those  without  the  TADSS. 

Changes  to  training  execution  will  usually  occur  in  response  to  deliberate  decisions,  for 
example  implementing  innovations  from  the  Simulation  in  Training  for  Advanced  Readiness 
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(SIMITAR)  program  (Krug  &  Pickell,  1996).  The  research  task  in  such  cases  is  straight- 
forward-identify  the  resources  and  performance  likely  to  be  affected  and  compare  the  trend  lines 
before  and  after  the  introduction  of  the  iimovation.  Sometimes,  however,  the  change  may  be 
gradual  (e.g.,  changes  in  ability  levels).  Then  researchers  have  a  chance  to  uncover  the  factors 
that  contribute  to  the  change  in  the  trend  line  (e.g.,  economic  conditions,  modifications  to 
recruiting). 

Since  the  trend  lines  document  natural  variation  without  the  random  assignment  required 
for  experimental  treatments,  it  will  usually  be  necessary  to  consider  whether  other  trends  might 
be  influencing  the  results.  Causal  models  and  regression  analysis  techniques  applied  to  multiple 
units  help  document  the  impact  of  potential  moderating  variables,  such  as  ability  level  and 
turbulence. 

Experimental  Explanation 

In  some  cases.  Army  planners  will  implement  experiments  in  which  units  are  assigned  at 
random  to  treatment  conditions  and  the  effects  of  the  treatments  are  documented.  For  example, 
plans  for  the  SIMITAR  initiative  included  experimental  and  control  battalions  in  rotations  to 
NTC  (Krug  &  Pickell,  1996).  Even  though  most  experimental  designs  do  not  depend  on 
maintaining  data  over  a  long  time  period,  the  database  can  support  such  experiments  by 
documenting  the  extent  to  which  the  training  intervention  was  implemented  (e.g.,  TADSS  usage) 
in  experimental  units  and  describe  comparable  training  programs  for  the  control  units. 

Research  Questions 

What  interventions  or  interruptions  have  an  impact  on  unit  performance?  This  is  the 
fimdamental  question  to  be  addressed  by  the  ARI  training  database.  Answering  it  requires  a 
substantial  commitment,  at  least  for  early  prototype  analyses.  Research  should  assess  the  impact 
of  planned  events  and  include  the  monitoring  of  trend  lines  to  detect  evidence  of  changes  to 
resources  or  performance  and  development  of  hy{K)theses  for  causes  of  those  changes. 

How  can  the  armor/mechanized  infantry  database  be  maintained  and  accessed?  Under 
current  conditions,  the  database  requires  a  great  deal  of  attention  to  update  information  on  current 
battalions  and  to  extend  the  number  of  battalions  covered.  Data  from  sources  such  as  the 
Standard  Installation/Division  Personnel  System  (SIDPERS),  COFT,  battle  rosters,  and  drill 
attendance  should  be  added  monthly  or  quarterly.  Those  sources  are  currently  in  a  paper-based 
format  and  must  be  entered  by  hand.  Research  should  be  directed  toward  developing  interfaces 
with  the  data  sources,  perhaps  through  the  Reserve  Component  Automation  System,  to  enable 
digital  transfer  of  data. 

Compiling  data  requires  substantial  support  from  units.  The  first  requirement  for 
securing  and  maintaining  that  support  is  to  ensure  that  units  obtain  direct  pay-off  from  the 
database.  During  the  initial  stages,  this  requires  frequent  coordination  between  ARI  researchers 
and  unit  trainers  to  provide  status  reports  and  prototype  analyses.  At  a  later  stage,  the  database 
would  be  most  useful  if  unit  trainers  could  retrieve  information  directly  in  a  user  friendly  format. 
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The  current  structure  for  retrieving  information  (Statistical  Package  for  the  Social  Sciences- 
SPSS)  requires  assistance  by  a  researcher.  The  ultimate  goal  should  be  access  through  SATS, 
but  interim  solutions  should  be  identified.  Research  should  consider  development  of  standard 
queries,  graphical  support,  and  Internet  access. 

A  further  requirement  for  securing  unit  support  for  the  database  is  to  protect  units  from 
comparisons  that  might  embarrass  their  leadership.  In  early  stages,  this  protection  requires 
monitoring  by  researchers.  When  units  have  direct  access,  the  database  will  require  seciority 
procedures  that  prevent  identification  of  other  individual  units. 

How  should  variables  be  aggregated  for  a  collective  training  database?  Much  of  the 
information  on  individual,  crew,  and  squad  level  characteristics  and  achievements  will  need  to  be 
aggregated  for  comparison  to  performance  indicators  for  higher  level  units.  For  example,  an 
analysis  of  the  relation  between  individual  ability  and  platoon  battlerun  performance  might 
aggregate  ability  measures  in  three  ways  (Ford  &  Keesling,  1994):  (a)  mean  Armed  Forces 
Qualification  Test  (AFQT)  of  crewmen,  (b)  percentage  of  high  ability  crewmen  (AFQT  Category 
1  and  2),  and  (c)  percentage  of  low  ability  crewmen  (AFQT  Category  3B  and  4).  Research 
would  consist  of  analyses  using  various  aggregation  schemes  and  provide  guidance  on  which  to 
apply  under  given  conditions. 


Distance  Training 

This  domain  includes  approaches  for  distributing  instruction  to  physically  dispersed 
students.  The  major  application  is  DL  for  individual  training,  typically  used  as  an  alternative  to 
resident  training.  An  emerging  application  is  Distributed  Interactive  Simulation  (DIS)  for 
collective  training. 

Distance  Learning 

ARI’s  support  of  DL  has  included  development  and  evaluation  of  content  for  the  full 
range  of  RC  individual  training:  (a)  professional  military  development  (e.g.,  leadership  training), 
(b)  reclassification  (initial  training  for  prior-service  personnel  whose  military  experience  does  not 
qualify  them  for  their  RC  duty  positions),  (c)  skill  progression  (preparation  for  more  advanced 
jobs  in  their  fields),  (d)  functional  needs  (e.g.,  counter-drug  operations),  and  (e)  special  skills 
(e.g.,  combat  lifesaver).  Most  of  the  current  work  is  being  conducted  in  support  of  DL  regional 
networks  established  by  the  National  Guard  Bureau  in  response  to  a  directive  from  the  National 
Defense  Authorization  Act  of  1995.  The  intent  is  to  establish  regional  networks  having  a  DL 
classroom  site  within  a  90-min  commute  for  all  ARNG  soldiers  (eventually  to  be  within  60  min). 
Each  site  will  have  an  audio  teleconferencing  station,  a  multi-media  personal  computer,  an 
interactive  graphic  tablet  and  software,  a  video  satellite  downlink,  a  videotape  player/recorder, 
and  a  35”  television  monitor.  Some  sites  will  also  include  a  network  of  12  multi-media  personal 
computers.  Continued  support  of  DL  for  RC  training  should  include  conversion  of  courses  to 
DL,  development  of  guidance  on  DL  delivery  methods  and  media,  assessment  of  cost 
effectiveness  of  DL,  and  development  of  guidance  on  DL  procedures. 
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Course  Conversion 


Army  leaders  have  identified  532  courses  that  are  appropriate  for  complete  or  partial 
conversion  to  a  DL  mode  of  delivery.  Much  of  the  impetus  for  the  planned  conversion  is  to 
increase  the  level  of  military  occupational  specialty  qualification  (MOSQ)  in  the  RC.  As 
summarized  in  Table  5,  about  twice  as  many  soldiers  who  require  MOSQ  training  could  be 
trained  each  year  when  those  courses  are  converted  to  DL  (Metzko,  Redding,  &  Fletcher,  1996). 

Table  5 


Requirements  for  Resident  Courses  for  Army  RC  Personnel 


Reserve 

Component 

Number  of  Courses 

Number  of  RC  Personnel  (FY95) 

Total 

Convertible  All 
or  in  Part  to  DL 

Needing  Courses 
Identified  as 
Convertible  to  67%  DL 

Likely  to  Receive 
Residential 
Instruction  in  FY 

ARNG 

1,600 

532 

12,606 

8,096 

USAR 

24,095 

10,104 

The  first  area  of  research  support  is  to  accelerate  the  rate  of  conversion  of  courses.  One 
way  to  support  the  conversion  of  courses  is  continued  assistance  with  monitoring  course 
conversion  and  evaluating  effectiveness  of  the  DL  network.  When  institutional  courses  are 
converted  for  ARNG  use,  they  are  restructured  to  fit  into  a  maximum  of  48, 4-hr  blocks  with  no 
more  than  15  days  of  accompanying  resident  training.  For  example,  the  course  for  MOS  93C 
(Air  Traffic  Controller)  is  being  conducted  with  21  students  fi'om  eight  states.  The  course 
includes  a  DL  portion  plus  2  weeks  of  resident  training  to  cover  tower  operations  and  radar 
(R.  Wisher,  personal  communication,  February  10,  1997).  Regulations  governing  DL  mandate 
that  such  comses  are  to  be  evaluated  as  they  are  implemented  (U.S.  Army  TRADOC,  1996). 
Such  evaluations  should  include  course  performance  indices,  student  opinions,  and  completion 
rates. 


In  addition  to  accelerating  the  rate  of  course  conversion,  research  should  also  expand  the 
scope  of  conversion.  Most  Army  courses  are  to  be  only  partially  converted  to  DL.  As  shown  in 
Table  6,  only  seven  courses  are  projected  to  be  delivered  in  a  completely  DL  mode  (Metzko  et 
al.,  1996).  In  contrast.  Air  Force  and  Navy  plans  call  for  conversion  of  all  courses  to  100%  DL. 
With  the  partial  conversion,  most  RC  personnel  in  DL  courses  v^dll  be  required  to  travel  to  a 
resident  training  site.  While  there  are  cost  implications  for  such  travel  (estimated  at  $800  per 
person),  the  major  concern  is  that  the  resident  requirement  increases  the  difficulty  for  RC 
personnel  to  receive  the  training:  15  days  is  less  of  a  problem  than  8  weeks,  but  it  is  still  a 
problem. 
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Table  6 


Planned  Conversion  of  Army  Courses  to  DL 


Percentage  of  Courses  Converted  to  DL* 

100 

94 

80 

70 

67 

50 

40 

33 

Number  of  Courses 

H 

B 

157 

177 

43 

3 

6 

135 

Percentage  of  532  Courses 

1 

1 

30 

33 

8 

1 

1 

25 

^Measured  in  hours  of  instruction 


Research  on  increasing  the  amount  of  DL  should  test  the  implied  assertion  that  resident 
training  is  needed  to  facilitate  learning  as  students  help  each  other.  For  example,  work  with 
Asynchronous  Computer  Conferencing  (ACC)  found  that  benefits  from  group  work  could  be 
achieved  in  a  DL  mode  (Phelps,  Ashworth,  Wells,  &  Hahn,  1991).  For  cases  in  which  resident 
training  is  justified  on  the  grounds  of  a  need  for  psychomotor  skill  training,  research  should 
assess  the  recommendation  by  the  Total  Force  DL  Action  Team  and  the  Guard/Reserve 
Subcommittee  to  provide  cognitive  skill  training  through  DL  and  conduct  skill  training  diuing 
monthly  inactive  duty  training  (IDT)  and  AT  (Metzko  et  al.,  1996). 

Methods  and  Media 

The  emphasis  on  video  transmission  capabilities  in  the  regional  demonstration  network 
and  the  wide-spread  use  of  video  teletraining  (VTT)  by  the  Army  and  other  Services  may  lead  to 
a  tendency  to  equate  DL  with  VTT.  In  fact.  Army  plans  for  the  DL  courseware  for  the  RC  calls 
for  only  about  10%  of  the  projected  hours  to  be  devoted  to  VTT,  with  about  half  of  the  projected 
hours  to  be  devoted  to  interactive  media,  such  as  computer-based  training  (CBT)  and  interactive 
videodisc  (IVD)/compact  disc  (CD).  This  emphasis  is  encouraging  since  interactive  courses  are 
consistently  found  to  be  more  effective  and  efficient  than  conventional  resident  courses  (Fletcher, 
1990;  Johnston  &  Fletcher,  1997). 

The  incorporation  of  media  other  than  VTT  enables  a  fundamental  decision  on  DL 
delivery:  whether  training  will  be  delivered  in  a  synchronous  or  asynchronous  mode. 
Synchronous  delivery  requires  simultaneous  “real  time”  interaction  between  the  instructor  and 
the  student.  In  contrast,  asynchronous  delivery  takes  advantage  of  interactive  media  and 
computer  conferencing  so  that  the  instructor  and  student  do  not  have  to  interact  concurrently. 
Asynchronous  delivery  is  especially  desirable  in  the  RC  because  it  allows  more  flexibility  for  RC 
personnel  to  schedule  training  aroimd  their  civilian  commitments.  The  TECHSELECT  decision 
aid  (Hagman  &  Dykstra,  1988)  guides  delivery  mode  decisions  based  on  considerations  of  the 
training  load  (e.g.,  fewer  than  10  participants  across  three  or  more  time  zones  argues  for 
asynchronous  delivery)  and  general  strategy  (e.g.,  lecture  argues  for  synchronous  delivery). 

Guidance  on  method  selection  should  be  extended  to  include  analysis  of  the  training 
content  tied  to  the  medium  best  suited  for  the  content.  Research  to  provide  that  guidance  should 
consider  emerging  videographic  and  audiographic  technologies  and  determine  the  optimal 
utilization  of  the  technologies  (Pugh,  Parchman,  &  Simpson,  1991). 
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Cost  Effectiveness 

Research  on  cost  effectiveness  should  document  effectiveness  of  DL  compared  to 
resident  training,  establish  costs  for  DL  conversion  and  delivery,  and  establish  costs  for  resident 
training. 

DL  effectiveness.  Research  on  DL  effectiveness  should  extend  beyond  determining 
whether  student  achievement  under  DL  is  at  least  equivalent  to  student  achievement  in  a  resident 
training  environment.  That  issue  is  well  established:  “The  efficacy  of  DL  technologies. .  .has 
been  demonstrated  in  the  military,  academia,  and  the  commercial  world  by  more  than  300 
studies”  (Metzko  et  al.,  1996,  p.  V-1).  As  described  in  the  section  on  DL  conversion, 
effectiveness  data  must  be  collected  as  part  of  the  tryout  of  new  instruction.  The  point  here  is 
that  the  design  of  the  required  effectiveness  studies  should  also  enable  an  assessment  of  the 
impact  of  variables  that  affect  the  quality  and  cost  of  DL,  for  example,  feedback,  quality  of 
materials,  variety  of  media,  and  class  size  (Hahn,  Ashworth,  Phelps,  Wells,  Richards,  & 

Daveline,  1991). 

As  an  illustration  of  how  assessment  of  DL  effectiveness  can  inform  decisions  on 
methods  and  media,  consider  two  recent  studies  of  variations  in  VTT.  The  Army  Teletraining 
Network  (TNET)  project  studied  strategies  for  delivering  leader  training  in  the  Basic 
Noncommissioned  Officer  Course  (BNCOC)  that  included  one-way  video,  two-way  audio  (IV, 

2 A)  and  two-way  video,  two-way  audio  (2V,  2 A).  Students  in  the  IV,  2A  condition  performed 
as  well  as  students  receiving  traditional  instruction  and  better  than  students  in  the  more 
expensive  2V,  2A  condition  (Wilson,  1992).  The  Army  Logistics  Management  College  (ALMC) 
study  compared  cost  per  student  in  resident  training  with  costs  for  training  with  on-site 
instructors  (save  56%),  analog  satellite  (save  70%),  and  compressed  digital  video  (save  79%) 
(Redding  &  Fletcher,  1994). 

DL  costs.  Research  on  cost  effectiveness  requires  credible  measures  of  the  costs  to 
convert  a  resident  course  to  DL.  When  the  Total  Force  DL  Action  Team  and  the  Guard/Reserve 
Subcommittee  collected  estimates  of  conversion  costs  from  the  Services,  the  range  of  estimates 
was  very  wide.  The  uncertainties  about  conversion  costs  are  illustrated  in  Table  7,  which  applies 
the  estimates  to  projected  Army  course  conversions  (Metzko  et  al.,  1996). 

The  400%  range  of  conversion  costs  in  Table  7  suggests  either  that  there  are  substantive 
differences  in  the  way  Services  implement  the  media  or  that  estimates  are  based  on  limited 
examples.  In  either  case,  more  accurate  conversion  costs  based  on  actual  experience  is  required. 
An  example  of  the  level  of  detail  required  is  documentation  of  the  new  lesson  plan,  new  student 
aids,  new  instructor  guide,  and  validation  of  lessons  (Hahn  et  al.,  1991).  Even  with  uniformly 
low  estimates,  conversion  is  a  significant  cost  that  should  be  given  more  support  in  DL  funding 
decisions.  Metzko  et  al.  (1996)  reported  that  about  5%  of  the  funds  expended  on  VTT  in  the 
Army  for  FY94  were  devoted  to  preparing  courseware;  less  than  10%  of  the  funds  allocated  for 
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Table  7 


Cost  to  Convert  Army  Resident  Course  Candidates  to  DL  Media 


Hours 

Cost  per  Hour  (CPH)^  ($) 

Cost  to  Convert  ($  thousand)) 

Low 

High 

Low 

High 

Print 

15,932 

1,300 

5,000 

20,712 

79,660 

Videotape 

10,572 

8,300 

17,000 

87,748 

179,724 

CBT 

22,575 

4,000 

14,000 

90,300 

316,050 

IVD/CD 

17,564 

4,000 

19,000 

70,356 

333,716 

VTT 

9,771 

260 

19,000 

2,540 

185,649 

Tol 

.al" 

271,656 

1,094,799 

CPHs  are  highs  and  lows  of  all  Services’  estimates,  except  for  VTT  (Navy  CPH  was  0). 
'’Does  not  include  Army  “other”  category,  about  8,500  hours  devoted  to  audio 
conferencing,  with  high  and  low  CPH  of  $10,000. 


the  regional  demonstration  network  in  FY95  were  designated  for  courseware.  Research  that 
identifies  the  scope  of  conversion  activities  and  establishes  credible  cost  projections  would 
support  argxmients  for  increased  funding  for  course  conversion. 

Costs  for  delivering  DL  should  also  be  documented.  For  ACC,  these  costs  include 
production  of  materials,  procurement  of  equipment,  training  of  instructors,  shipping  of  materials, 
and  operating  and  support  costs  (e.g.,  instructor  salaries).  An  innovation  in  the  ACC  cost 
effectiveness  analysis  is  the  development  of  a  rubric  of  cost  per  50  graduates,  which,  when 
compared  to  a  similar  figure  for  resident  training,  accounts  for  through-put  differences  (Hahn  et 
al.,  1991). 

Cost  effectiveness  comparisons  between  DL  and  resident  training  obyiously  require  data 
on  resident  costs.  Direct  costs  are  easy  to  calculate:  travel  and  per  diem,  instructor  costs,  and 
training  material.  More  difficult  calculations  would  consider  the  marginal  infi-astructure  costs, 
such  as  training  developers,  training  equipment,  and  school  operations.  The  ultimate  benefits  for 
RC  training  relate  to  opportunity  costs  and  increases  in  readiness. 

Administrative  Guidance 

As  experience  with  DL  accumulates,  issues  concerning  the  relation  between  the  way 
courses  are  administered  and  their  effectiveness  will  arise.  Three  such  issues  are  instructor 
selection,  control  practices  that  facilitate  completion,  and  reimbursement  of  DL  participants. 

Instructor  selection.  The  characteristics  and  experience  that  make  an  instructor  effective 
in  a  resident  training  environment  may  not  apply  for  the  DL  environment.  For  asynchronous 
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training,  the  instructor’s  role  is  redefined.  In  resident  training,  verbal  interactions  are 
monopolized  by  the  instructor  (60%  to  80%).  In  contrast,  the  instructor  is  less  verbal  in  an 
asynchronous  mode;  for  example,  only  10%  to  15%  of  class  dialogue  in  ACC  is  contributed  by 
the  instructor  (Davie  &  Wells,  1992).  The  revised  role  requires  instructors  to  encourage  student 
participation  through  positive  reinforcement  or  by  designing  assignments  that  provoke 
participation.  In  synchronous  DL,  the  instructor’s  role  is  similar  to  the  role  in  resident  training 
but  may  be  more  demanding.  For  example,  a  review  of  VTT  literature  for  the  Navy  concluded 
that  students  expect  instructors  to  be  equivalent  to  moderators  in  programs  on  commercial 
television-dynamic  actors  with  a  sense  of  humor-as  well  as  SMEs  (Bailey,  Sheppe,  Kodak, 
Kruger,  &  Smith,  1989). 

Facilitating  completion.  A  consistent  advantage  of  resident  training  over  DL  is  that 
more  students  complete  training  in  a  resident  environment.  Causes  for  that  advantage  include 
fewer  disruptions  for  students  (e.g.,  from  civilian  job  demands)  and  the  shorter  schedule  for 
resident  training,  which  reduces  the  chance  that  crises  will  force  the  student  to  drop  out.  There 
are,  however,  student  motivation  variables  that  increase  completion  of  asynchronous  DL:  (a) 
high  student-instructor  interaction,  (b)  a  flexible  but  firm  schedule,  (c)  group  activities,  and  (d) 
incentives  and  sanctions  (e.g.,  bonus  points  for  early  lesson  completion  and  deductions  for  being 
late).  For  synchronous  training,  it  may  be  possible  to  shorten  the  time  fi-ame  for  the  course  by 
lengthening  the  VTT  sessions  beyond  4  hours  within  a  24-hour  period.  For  example,  the  Army 
TNET  Study  found  that  students  in  the  2V,  2A  condition  taught  over  7  days  performed  as  well  as 
students  trained  over  4  months  (Wilson,  1992). 

Reimbursement.  The  third  administrative  area  that  affects  the  success  of  DL  concerns 
reimbursement  of  students.  Enlisted  personnel  and  officers  in  resident  training  receive  their 
active  duty  pay  while  at  the  training  site.  In  a  DL  mode,  only  enlisted  personnel  receive  active 
duty  pay.  For  synchronous  training,  they  can  be  paid  based  on  the  length  of  the  DL  session.  For 
asynchronous  training,  the  question  is  whether  reimbursement  should  be  based  on  completion  of 
course  requirements  or  total  time  devoted  to  DL. 

Research  Questions 

Although  the  effectiveness  of  DL  has  been  well  documented  through  case  studies  (Wells, 

1990) ,  there  has  been  comparatively  little  effort  to  design  studies  that  extend  the  findings  to  other 
DL  efforts.  Notable  exceptions  to  this  tendency  are  the  Army’s  work  with  ACC  (Hahn  et  al., 

1991)  and  the  Navy’s  work  with  VTT  (e.g.,  Simpson,  Pugh,  &  Parchman,  1993).  These 
exceptions  demonstrate  that  studies  whose  immediate  purpose  is  quality  assurance  for  DL 
courseware  also  provide  opportunities  to  introduce  systematic  variation  in  the  design  to  address 
broader  questions.  Four  of  these  questions  are  recommended  below. 

What  is  the  most  cost-effective  combination  of  media  in  DL?  The  answer  will  depend  on 
characteristics  of  the  tasks,  the  training  audience,  and  the  delivery  environment.  The  anticipated 
product  is  a  computerized  decision  support  tool,  similar  to  TECHSELECT  (Hagman  &  Dykstra, 
1988),  with  an  emphasis  on  taking  advantage  of  interactive  media  in  course  conversions. 
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What  types  of  interactivity  are  required  in  DL?  Research  related  to  this  question  will 
identify  the  types  of  students  and  tasks  that  require  interactive  presentation  and  the  types  of 
interactions  required  (e.g.,  whether  with  the  instructor  or  with  other  students).  For  asynchronous 
training,  variables  should  include  frequency  and  type  of  group  activities  and  frequency  of 
contacts  with  the  instructor.  For  synchronous  training,  the  major  variable  will  be  video  and 
audio  alternatives:  in  descending  order  of  cost,  2V/2A,  1V/2A,  1 V/1  A,  and  audiographics  (two- 
way  audio  and  still-frame  graphics  that  can  be  annotated).  The  scope  of  the  question  also 
includes  documenting  the  benefits  of  resident  requirements  for  part  of  the  course  (rather  than  full 
DL). 


What  skills  are  required  for  instructors  to  be  effective  in  DL?  For  synchronous 
delivery,  the  question  should  address  the  importance  of  presentation  skills  of  the  VTT  instructor 
and  characteristics  for  effective  on-site  facilitators.  For  asynchronous  delivery,  the  question 
should  consider  skills  to  encourage  participation.  The  answers  should  guide  instructor  training 
as  well  as  selection  of  instructors. 

What  savings  can  be  anticipated from  DL?  Answering  this  question  involves  collecting 
and  aggregating  data  on  infrastructure  costs  for  both  resident  and  DL  training,  travel  and  per 
diem  costs,  resident  training  costs,  costs  to  convert  conventional  courses,  and  expected 
improvements  in  training  efficiency  and  personnel  readiness  (Metzko  et  al.,  1996).  Since  the 
content  of  much  training  applies  to  the  Total  Force,  savings  should  be  projected  for  AC 
personnel  and  civilians  as  well  for  RC  personnel. 

Distributed  Interactive  Simulations  (DIS) 

This  section  describes  research  on  extending  the  application  of  technology  to  support  DL 
of  individuals  to  the  support  of  collective  training  of  dispersed  units.  The  research  should 
consider  DIS  possible  with  cmrent  technology  and  ways  to  refine  distributed  interactive  training 
technologies  for  more  realistic  interactions. 

Current  DIS 

The  “virtual  classrooms”  established  to  support  individual  DL  also  can  be  used  for  DIS 
for  collective  training.  Work  by  ARJ  has  demonstrated  that  a  brigade-level  CPX  in  DIS  can 
achieve  comparable  results  with  substantial  savings  over  a  field  CPX.  The  demonstration  used 
relatively  imsophisticated  long  distance  technology  (e.g.,  slow-scan  television  transceivers, 
facsimile  machines,  and  commercial  telephones)  and  the  Computer  Assisted  Map  Maneuver 
Simulation  (CAMMS)  as  the  battlefield  simulator  (Smith,  Hagman,  &  Bowne,  1987).  Using  the 
“virtual  classroom”  for  video  teleconferencing  (VTC)  among  staffs  and  the  more  advanced 
Battalion  Brigade  Simulator  (BBS)  should  allow  current  RC  battalions,  brigades,  and  divisions  to 
achieve  comparable  or  improved  results.  In  fact,  an  ARNG  brigade  commander  who  was 
interviewed  for  this  project  considered  VTC  to  be  “the  most  important  step  in  ARNG  training 
since  the  1950s,”  specifically  citing  the  capability  of  conducting  staff  training  for  all  battalions  in 
the  state  without  travel  requirements  (COL  Youngman,  personal  communication,  December  12, 


38 


1996).  Research  on  the  effectiveness  of  DIS  for  staff  training  would  demonstrate  the  savings  to 
be  anticipated  and  establish  procedures  to  implement  the  training. 

Refined  DIS 

As  part  of  SIMITAR,  ARNG  trainers  are  developing  procedures  to  link  Janus  in  a  wide- 
area  network  (WAN)  to  support  a  CPX.  The  advantage  of  the  WAN  is  that  all  echelons  down  to 
battalion  are  required  to  exercise  the  command  and  control  fimction.  Since  current  software  cannot 
support  the  WAN  application,  interim  training  will  be  conducted  using  parallel  Janus  sets.  When 
constructive  tactical  engagement  simulations  (TES)  (e.g.,  Janus  and  BBS)  can  be  conducted  in  a 
WAN,  it  will  be  possible  to  conduct  virtual  TES,  such  as  SIMNET  and  CCTT,  in  a  DIS  mode. 

Research  Questions 

What  savings  can  be  achieved  by  conducting  collective  training  in  a  DIS  mode?  This 
question  concerns  current  DIS  technology.  The  first  need  is  to  determine  whether  constructive 
simulations  can  be  delivered  effectively  using  the  current  VTC  capability  in  the  ARNG  and 
current  constructive  simulations.  Assuming  that  the  training  will  be  effective  and  efficient, 
research  should  focus  on  development  of  scenarios  and  TSPs  for  the  training. 

How  can  TES  systems  be  integrated  to  support  training  in  a  DIS  mode?  Research 
related  to  this  question  requires  the  design,  development,  and  evaluation  of  software  that  enables 
systems  from  geographically  distributed  sites  to  interact  in  real  time. 

Concluding  Comments  on  the  Training  Research  Domain 

The  dominant  theme  in  the  preceding  discussion  of  training  research  has  been  optimizing 
TADSS  and  DIS  to  increase  training  time  and  effectiveness.  Given  the  constraints  of  the  RC 
environment,  these  are  important  research  areas.  But  simply  having  a  TADSS  does  not  create 
effective  training.  They  must  be  appropriately  designed  and  implemented.  The  research 
suggestions  above  have  been  offered  to  address  both  of  these  issues.  Clearly,  none  of  the 
training  problems  faced  by  the  RC  is  simple.  Tasks  at  the  individual  level  are  complex  and 
varied  so  soldiers  must  work  hard  to  keep  their  proficiency  levels  up.  Collective  tasks  are  even 
more  complex  and  difficult  to  maintain.  Any  loss  of  training  efficiency  is  difficult  to  overcome 
in  the  RC.  Therefore,  even  the  seemingly  mundane  questions  that  have  been  addressed  over  and 
over  by  applied  training  researchers  must  continue  to  be  addressed.  ARI  must  continue  its  close 
involvement  in  evaluating  developing  TADSS.  It  is  only  though  close  involvement  in  the 
development  process  that  researchers  can  begin  to  understand  both  the  potential  and  the 
limitations  of  new  devices. 

Rigorous  empirical  research  in  the  RC  environment  (or  the  active  Army  for  that  matter)  is 
however,  very  difficult  to  conduct.  Researchers  are  often  simply  not  given  the  control  required.  On 
the  other  hand,  researchers  can  have  an  impact  on  training  effectiveness  though  less  structured 
means.  Action  research  models,  which  encourage  exploration  and  demand  continuous  monitoring 
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of  system  functioning,  provide  an  alternative  method  for  learning  about  the  utility  and  effective  use 
of  promising  TADSS.  Unfortunately,  such  approaches  do  not  lead  to  quick  answers.  Furthermore, 
action  research  requires  that  training  managers  treat  implementation  decisions  as  experiments  to  be 
followed  closely  and  adjusted  as  necessary.  This  model  may  be  counterintuitive  to  the  culturally 
reinforced  decisive  nature  of  the  active  army,  and  it  is  contrary  to  the  rigorous  experimental  design 
tradition.  On  the  other  hand,  the  model  may  be  more  acceptable  to  RC  officers  who  are 
increasingly  being  exposed  to  total  quality  management  (TQM)  methods  of  action  research. 

TADSS  are  increasingly  complex;  consequently  the  chances  of  implementing  them  optimally, 
without  some  extended  period  of  experimentation,  is  increasingly  unlikely. 

Perhaps  not  surprisingly,  project  interviews  did  not  imcover  an  abundance  of  exciting 
new  ideas  for  training  research.  In  our  experience,  however,  non-researchers  tend  not  to 
spontaneously  think  of  conducting  research,  even  action  research,  as  a  problem-solving  approach 
to  organi2ational  issues.  That  is  not  their  aggressive  cognitive  "style"  (Sternberg  &  Grigorenko, 
1997).  On  the  other  hand,  we  will  take  the  liberty  of  suggesting  a  few  ideas  that  are  at  least  a 
step  outside  the  programs  that  ARI  has  been  pursuing. 

Research  Questions 

What  are  the  effective  training  delivery  techniques?  Recent  research  has  given  little 
detailed  attention  to  training  delivery  techniques.  As  observers  of  hours  of  AC  and  RC  training 
implementations,  we  have  seen  wide  variations  in  delivery  methods,  even  among  instructors 
using  the  same  TADSS.  For  example,  Graham  and  Smith  (1991)  noted  the  variety  of  techniques 
that  UCOFT  instructors  used  to  teach  and  coach  tank  guimery  crewing,  even  though  they  were  all 
following  the  same  basic  UCOFT  matrix  progressions.  Most  TADDS  tend  to  have  operators  or 
instructors  that,  through  their  interactions  with  and  instructions  to  the  soldiers,  have  an  influence 
on  the  quality  of  the  instruction.  ARI  might  begin  this  research  area  by  offering  to  play  a 
technical  assistance  role  for  some  particular  TADDS  (e.g.,  mobile  SIMNET,  AFIST,  or 
UCOFT).  In  this  capacity,  researchers  could  develop  two  or  three  alternative  techniques  for 
instructors  to  use  and  then  build  a  method  for  obtaining  feedback  on  how  the  techniques  worked. 
An  action  research  model,  making  the  work  a  collaborative  effort  with  the  instructors,  would  be 
suitable.  We  suggest  that  ARI  ground  its  alternative  recommends  on  recent  cognitive 
psychology  models  of  knowledge  acquisition  (see  Hoffinan,  1997,  for  an  example  related  to 
brigade  staff  training). 

What  is  the  training  delivery  capability  of  the  Internet?  The  Internet  is  here  to  stay  and 
provides  a  potentially  powerful  vehicle  for  training  delivery.  This  second  new  research  area  is 
related  to  the  above  suggestion  in  that  both  pay  attention  to  the  details  of  instructional  delivery. 
The  hypermedia  capabilities  of  the  Internet  naturally  lead  to  research  concerning  alternative 
structures  and  organizations  for  delivering  information  in  a  way  the  best  facilitates  learners 
gaining  familiarity  with  important  concepts,  and  then  being  stimulated  into  the  reflective  thought 
needed  to  create  organized  conceptualizations  of  the  topics. 

Are  there  alternative  evaluation  models?  This  suggestion  is  stimulated  by  our  previous 
observation  on  the  difficulty  of  conducting  research  according  to  the  standards  set  by  Boldovici 
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(1987)  and  Hoffman  and  Morrison  (1992)  and  by  recent  thinking  on  evaluation  models  by 
Alliger,  Taimenbaum,  Bennet,  Traver,  and  Shetland  (1997)  and  Kraiger  et  al.  (1997).  Neither 
what  Boldovici  or  Hoffman  and  Morrison  have  proposed  are  very  realistic  in  light  of  the  climate 
of  the  RC,  AC,  or  anywhere  else  for  that  matter.  Alliger  et  al.  and  Kraiger  et  al.  are  envisioning 
alternative  views.  Additional  thinking  for  the  RC  environment,  however,  is  recommended. 
Although  a  step  removed  from  immediate  results  for  the  RC,  ARI  should  consider  spending  time 
reflecting  on  feasible  research  strategies  for  real-world  training  questions.  By  all  appearances,  as 
much  training  evaluation  work  occurs  in  the  military  as  in  private  industry.  Therefore,  ARI  is 
poised  to  take  a  leadership  role  in  this  area.  Program  evaluation  theorists  have  been  wrestling 
with  these  issues  since  D.T.  Campbell  and  Stanley  (1966),  and  there  is  a  rich  literature  base  for 
stimulating  thought.  The  kinds  of  social  programs  they  are  often  attempting  to  evaluate  are  every 
bit  as  complex  as  any  training  program  in  the  Army.  The  action  research  ideas,  which  are  part  of 
the  organizational  development  and  TQM  literature,  are  also  applicable.  This  literature 
recognizes  and  tries  to  accommodate  the  control  issues  that  continually  confront  evaluation 
research. 


MANPOWER  AND  PERSONNEL  RESEARCH  DOMAINS 

In  this  part  of  the  report,  attention  shifts  from  training,  an  area  in  which  ARI  has 
contributed  a  great  deal  of  R&D  in  support  of  the  RC,  to  manpower  and  personnel,  areas  in 
which  ARI  RC-specific  research  has  been  more  limited.  It  is  also  true,  however,  that  there  are 
rich  research  literatures  related  to  several  of  the  domains  discussed  in  the  following  sections  that, 
when  combined  with  the  RC  work  ARI  has  conducted,  yield  a  substantial  foundation  for  future 
research. 


Recruiting 

Historically,  recruiting  into  the  RC  has  not  been  a  priority  concern.  And  during  the  high 
point  of  the  drawndown  in  the  AC,  both  the  Guard  and  the  Reserves  benefited  from  the  unusually 
high  numbers  of  prior  service  individuals  that  came  into  the  recruiting  market  (Kirby  &  Buddin, 
1996).  But  the  recruiting  environment  is  rapidly  changing,  making  recruiting  for  the  RC  more  of 
a  challenge.  For  example,  there  is  a  steadily  decreasing  propensity  to  serve  in  the  armed  forces, 
particularly  among  black  youth  (OASD,  1996).  Guard  persoimel  are  particularly  concerned  that 
those  who  do  serve  are  increasingly  motivated  by  incentives  other  than  patriotism,  making  their 
commitment  to  the  service  more  tenuous.  And  even  though  the  U.S.  is  not  at  war,  the  increasing 
likelihood  that  soldiers  will  be  deployed  for  a  variety  of  other  missions  can  also  be  expected  to 
impact  recruiting  success.  The  types  of  missions  (e.g.,  peacekeeping  in  inhospitable  locations) 
and  the  number  of  missions  are  both  potential  issues  here.  Another  anticipated  change  that  is 
likely  to  affect  recruiting  strategies  is  the  increasing  percentage  of  combat  service  and  combat 
service  support  positions  relative  to  combat  positions.  Such  changes  in  responsibilities  will 
influence  the  types  of  persons  interested  in  serving. 

In  contrast  to  the  AC,  deployments  are  often  on  a  volunteer  basis  for  Guard  and  Reserve 
personnel.  Accordingly,  we  must  be  concerned  about  recruiting  for  such  special  assignments,  as 
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well  as  recruiting  into  the  force.  Recent  experiences  with  such  deployments  indicate  that  there  is 
much  to  be  learned  about  improving  the  recruiting  processes  associated  with  them. 

The  Guard  and  Reserves  have  each  taken  measures  on  their  own  to  address  concerns  in 
the  recruiting  arena.  For  example,  the  Office  of  the  Chief  of  the  Army  Reserve  (OCAR)  has 
commissioned  several  studies  related  to  recruiting  and  retention,  some  of  which  are  described 
below.  The  Guard  has  taken  a  more  operational  approach  to  dealing  with  the  situation  by 
consolidating  recruiter  and  retention  NCO  career  fields  (OOE  and  79D)  into  a  new  “Strength 
Management”  NCO  career  field.  This  effort  to  explicitly  link  recruiting  and  retention  strategies 
is  sensible,  since  efforts  to  increase  recruiting  success  could  ultimately  lead  to  increased  attrition, 
trading  off  short  term  gains  for  long  term  losses. 

Additional  research  has  been  conducted  by  agencies  other  than  Department  of  Army 
(DA),  most  particularly  DoD.  This  work  has  been  related  to  a  number  of  areas,  including 
enlistment  decision-making,  projections  of  youth  market  characteristics,  and  recruiter  attitudes 
and  effectiveness.  A  large  proportion  of  this  work  has  been  (or  is  being)  performed  by  the  Rand 
Corporation.  Several  related  pieces  of  research  have  also  been  conducted  by  researchers  at  the 
Naval  Postgraduate  School,  primarily  for  the  U.S.  Army  Recruiting  Command  (USAREC), 

Much  of  the  literature  referred  to  in  this  section  is  described  in  a  report  available  from  the 
Defense  Manpower  Data  Center  (DMDC),  Compendium  of  Annotated  Military  Manpower 
Market  Research  Abstracts  (Barnes,  Dempsey,  Gaskins,  Knapp,  Lerro,  &  Schroyer,  1991). 
Though  somewhat  dated  at  this  point,  this  report  includes  a  comprehensive  collection  of  detailed 
abstracts  and  evaluations  of  military  marketing  research  conducted  prior  to  1990. 

Army  Research 

Perhaps  the  most  extensive  study  related  to  Army  recruiting  is  the  New  Recruit  Survey 
(NRS)  program  initiated  in  the  early  1980's  by  ARI  in  support  of  USAREC  (e.g.,  Levine,  1987; 
Westat,  1986a,b,c).  This  program  grew  out  of  the  1979  DoD  Survey  of  Personnel  Entering 
Military  Service  (Doering,  Grissmer,  &  Morse,  1980)  and  the  1982  DA  Survey  of  Personnel 
Entering  the  Army  (Elig,  Johnson,  Gade,  &  Hertzbach,  1984).  In  the  late  1980's,  USAREC 
assumed  responsibility  for  administering  the  NRS  and  analyzing  the  data. 

Analyses  of  NRS  data  have  been  rather  sporadically  reported,  and  it  is  not  known  how 
extensively  USAREC  cvurently  makes  use  of  the  data.  ARI  researchers  Pliske,  Elig,  and  Johnson 
(1986)  reported  analyses  of  1982/1983  NRS  data  from  AC  recruits  in  which  they  examined  a 
model  of  enlistment  decision  making.  The  Naval  Postgraduate  School  (Gray,  1987)  used  1985 
AC  NRS  data  to  examine  differences  between  high  and  low  aptitude  new  recruits  in  enlistment 
motivations.  High  aptitude  recruits  were  more  likely  to  be  influenced  by  bonuses,  the  two-year 
enlistment  option,  and  educational  benefits  compared  to  lower  aptitude  recruits.  Lower  aptitude 
recruits  were  motivated  by  opportunities  for  skill  training,  earning  more  money,  retirement,  and 
fringe  benefits.  Similar  findings  were  reported  for  female  recruits. 
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There  have  been  some  published  analyses  of  NRS  data  collected  from  RC  recruits.  An 
ARJ  researcher.  Dale  (1989),  used  NRS  data  collected  from  new  RC  recruits  in  1982  to  study 
attrition  from  the  RC.  This  research  was  described  in  our  Volume  1  report.  Simple  tabulations 
of  RC  NRS  data  collected  in  1985  have  been  published  (Westat,  1986a,b,c)  and  two  reports  of 
analyses  of  1987  NRS  data  on  RC  recruits  conducted  for  USAREC  are  also  available  (Gorman  & 
Thomas,  1989;  Schroyer,  Gaskins,  &  Waters,  1988).  Gorman  and  Thomas  looked  at  data  from 
1,364  male  and  397  female  non-prior  service  USAR  recruits.  They  found  that  high  aptitude 
recruits  were  motivated  more  by  money  and  job  considerations,  whereas  lower  aptitude  recruits 
were  motivated  more  by  patriotism  and  family  tradition.  In  another  analysis  of  1987  NRS  data, 
Schroyer  et  al.  compared  enlistment  motivations  of  AC,  Guard,  and  Reserve  recruits.  Of  these 
three  groups,  Army  Reserve  recruits  were  the  most  influenced  by  money  for  college.  They  also 
had  the  highest  aptitude  scores  and  educational  goals.  AC  recruits  were  more  highly  interested  in 
training  opportunities  than  RC  recruits. 

Another  relatively  extensive  ARI  research  effort  to  study  youths’  decision-making  with 
regard  to  joining  the  military  (both  enlisted  and  officer  corps)  was  the  Army  Communications 
Objectives  Measurement  Systems  (ACOMS)  survey  program  (e.g.,  Nieva,  Gaertner,  Elig,  & 
Benedict,  1988).  Data  were  collected  via  computer  assisted  telephone  interviews  from  young 
people  16-24  years  old  during  1986-1987.  Analyses  conducted  on  these  data  explored  a  variety 
of  areas,  including  brand  differentiation  among  Army  components  and  the  other  military 
services,  media  habits,  recall  of  Army  advertising,  and  career  choice  behaviors.  For  example, 
results  showed  that  the  USAR  was  relatively  undifferentiated  from  the  Army  AC,  whereas  the 
ARNG  was  more  clearly  distinguishable  to  youth  as  a  different  career  choice. 

In  another  project  conducted  for  USAREC,  ARI  studied  the  enlistment  propensities  of 
civilian  registered  nurses  and  nursing  students  (Ramsberger,  Barnes,  DiFazio,  &  Tiggle,  1994). 
The  Army  Nurse  Corp  is  more  than  three-quarters  RC  officers,  and  adequately  maintaining 
manpower  levels  is  a  constant  challenge  for  recruiters.  This  research  showed  that  propensity  to 
serve  was  generally  low,  and  survey  respondents  indicated  that  they  would  have  been  more 
inclined  to  join  prior  to  Operations  Desert  Shield  and  Storm  (ODS).  Nurses  with  a  relatively 
high  propensity  for  the  RC  had  a  greater  interest  in  research,  desire  to  attend  specialty  courses, 
and  had  higher  incomes  compared  to  those  with  lower  propensity  to  serve  in  the  RC. 

In  1995,  Amerind,  Inc.  conducted  several  studies  of  special  enlisted  populations  within 
the  USAR,  including  soldiers  early  in  their  obligation  (i.e.,  less  than  30  months  since  first  drill). 
In  this  study,  survey  data  were  collected  from  a  random  sample  of  147  reservists  with  less  than 
37  months  as  a  member  of  a  drill  unit  (Amerind,  Inc.,  1996b).  According  to  these  respondents, 
the  most  important  reason  for  enlisting  was  to  gain  GI  bill  benefits  and  another  consistently 
important  reason  was  to  gain  self-discipline.  The  “other”  category  was  also  heavily  endorsed  as 
being  important.  This  category  included  responses  such  as  the  following;  to  learn  a  skill, 
adventure,  to  be  proud.  Almost  half  of  the  respondents  reported  receiving  at  least  one  signing 
bonus. 


Amerind,  Inc.  (1996a)  also  conducted  a  survey  of  physicians  in  the  Reserve  to  examine 
recruiting  and  retention  issues  specific  to  this  population  of  officers.  A  total  of  835  physicians 
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responded  to  the  survey.  Almost  half  of  the  respondents  had  no  military  experience  prior  to  their 
service  in  the  USAR.  The  top  rated  reasons  for  joining  the  Reserve  were  service  to  country, 
professional  satisfaction,  and  change  of  pace.  Financial  incentives  were  more  influential  among 
younger  physicians  than  among  the  older  physicians. 

Previously,  the  OCAR  had  commissioned  a  study  to  examine  enlistment  motivations  of 
recruits  both  before  and  after  ODS  (Griffith  &  Perry,  1993;  Perry,  Griffith,  &  White,  1991). 
Surveys  were  administered  to  representative  samples  of  enlisted  reservists  in  1990  and  1991. 
Major  considerations  in  the  enlistment  decision  across  both  samples  were  wanting  to  experience 
the  military,  personal  development,  pay/benefits,  and  job/career  development.  Wanting  to 
experience  the  military  was  a  more  important  consideration  after  ODS  than  before,  whereas  the 
importance  of  personal  development  decreased. 

The  Naval  Postgraduate  School  has  also  conducted  a  number  of  different  studies  looking 
at  recruiting  issues  in  the  USAR.  These  studies  have  looked  at  issues  such  as  recruiting  females, 
using  Youth  Attitude  Tracking  Survey  (YATS)  data  to  examine  propensity  to  join,  effectiveness 
of  enlistment  incentives,  impact  of  commuting  distance  on  recruiting,  and  effects  of  population 
migration  patterns  on  recruiting. 

Other  Military  Research 

The  Office  of  the  Assistant  Secretary  of  Defense  (OASD)  has  sponsored  a  number  of 
efforts  to  assist  the  services  with  their  recruiting  efforts.  OASD  sponsors  YATS,  a  periodic 
survey  of  a  national  sample  of  youth  that  monitors  propensity  to  serve,  as  well  as  research  that 
uses  the  YATS  as  its  primary  data  source.  In  particular,  Rand  has  published  a  number  of 
analyses  of  YATS  data  that  explore  youths’  decision-making  processes  with  regard  to  career 
intentions  (e.g.,  Orvis,  Gahart,  &  Hosek,  1989).  A  couple  of  these  studies  have  focused 
specifically  on  recruiting  for  the  RC.  OASD  has  also  sponsored  periodic  surveys  of  AC  and  RC 
recruiters  to  gather  data  on  difficulties  they  experience  recruiting. 

Recruiting  for  Voluntary  Mobilizations 

ARI  research  conducted  to  examine  experiences  associated  with  the  use  of  RC  personnel  to 
support  a  rotation  of  the  Multinational  Force  Observers  (MFO)  mission  in  the  Sinai  (Phelps  &  Farr, 
1996)  was  described  in  our  Volume  1  report.  The  research  identified  a  number  of  ways  in  which 
the  mobilization  effort  presented  a  significant  challenge  to  the  Army  and  to  individual  soldiers  and 
officers.  Although  the  mission  was  met,  in  terms  of  the  numbers  and  quality  of  soldiers  deployed, 
considerable  effort  was  required  to  identify  replacements  for  individuals  who  initially  volunteered 
for  the  mission  but  then  subsequently  declined  to  participate.  Presumably,  the  no-show  problem 
could  be  reduced  by  examining  more  closely  and  alleviating  the  problems  soldiers  experienced 
during  the  mobilization  process.  The  problems  identified  in  ARI’s  MFO  study  were  conflicting 
needs  from  family,  job,  and  school.  There  was  also  evidence  that  inadequate  communication  with 
volunteers  throughout  the  mobilization  process  contributed  to  the  “no-show”  phenomenon. 
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Although  not  specifically  designed  to  examine  recruiting  problems,  both  the  Guard  and 
the  Reserve  have  explored  issues  associated  with  a  number  of  mobilization  efforts.  For  example, 
Amerind  conducted  a  series  of  focus  groups  of  Reserve  participants  in  Operation  Joint  Endeavor. 
Their  findings  were  very  similar  to  the  MFO  results  described  above.  The  National  Guard 
Bureau  published  an  After  Action  Report  for  ODS  (undated).  It  cited  the  need  for  refinement  of 
federalization  procedures  as  a  key  to  improved  mobilization  processes,  as  there  was  considerable 
confusion  regarding  federalization  status  and  requirements. 

Research  Questions 

Several  significant  research  questions  can  be  drawn  fi-om  our  analysis  of  the  current  RC 
recruiting  environment,  the  existing  literature,  and  our  interviews  with  RC  and  ARI  persoimel; 

What  strategies  for  recruiting  RC  enlisted  personnel  are  most  effective?  A  starting 
point  would  be  to  look  to  the  more  general  literature  (i.e.,  related  to  the  AC  and  other  services) 
on  enlistment  decision-making  to  take  advantage  of  the  considerable  work  that  has  been  done  to 
imderstand  and  model  the  enlistment  decision  (e.g.,  identifying  critical  influencers  and  decision 
criteria).  Though  the  models  might  require  adjustment  to  better  reflect  the  elements  associated 
with  the  decision  to  serve  in  the  RC,  the  basic  decision  process  should  be  similar. 

Another  potentially  valuable  line  of  research  is  to  assess  the  impact  of  state-level 
initiatives  for  recruiting  ARNG  personnel.  For  example,  at  least  two  states  have  established 
tuition-firee  incentives  for  people  who  enlist  in  the  ARNG.  Documenting  the  procedures  and 
assessing  the  effectiveness  of  such  incentives  would  enhance  the  program  in  those  states  and 
facilitate  similar  programs  in  other  states  (COL  Gilbert  &  COL  Shelley,  personal 
communication,  December  17, 1996). 

An  efficient  way  for  the  RC  to  gain  access  to  current  and  historical  information  about 
RC-related  enlistment  decision  making  would  be  to  develop  partnerships  with  USAREC  to  gain 
access  to  data  currently  being  collected  from  new  RC  recruits.  Such  a  partnership  could  also  lead 
to  inclusion  of  items  on  the  NRS  survey  that  would  be  of  particular  interest  to  the  RC.  A  related 
strategy  would  be  to  facilitate  collaboration  between  contract  researchers  who  work  for  the 
OCAR  and  USAREC  and  ARI  researchers,  all  of  whom  are  trying  to  leam  about  related 
enlistment  issues.  Given  DoD’s  clear  interest  in  recruiting  issues  across  the  services,  it  is  likely 
that  OASD  would  be  interested  in  supporting  and  participating  in  such  collaborative  efforts. 

What  strategies  for  recruiting  officers,  including  those  with  specialized  skills  (e.g., 
physicians,  engineers),  are  most  effective?  Historically,  the  process  for  routing  officers  into  the 
RC  fi-om  the  traditional  commissioning  sources  (i.e..  Reserve  Officer  Training  Corps  (ROTC), 
Academy,  Officer  Training  School)  has  meant  that  most  officers  must  be  recruited  as  they  depart 
the  AC  or  fi-om  the  pool  of  individuals  qualified  for  direct  commissions  (e.g.,  physicians, 
engineers).  So  research  has  focused,  and  should  probably  continue  to  focus,  on  these  types  of 
positions.  As  the  AC  downsizes,  however,  it  may  be  that  there  will  be  a  larger  pool  of  high  quality 
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ROTC  officer  candidates  available  to  the  RC.  At  a  minimum,  RC  units  should  be  encouraged  to 
maintain  close  contacts  with  ROTC  programs  to  assure  access  to  newly  available  junior  ofiScers. 

As  discussed  above,  ARI  has  already  conducted  some  research  related  to  nurse  recruiting 
and  the  OCAR  has  sponsored  research  on  physicians.  A  next  step  here  would  be  determine  the 
extent  to  which  the  OCAR  would  support  collaborative  studies  with  ARI,  and  to  determine 
priorities  for  which  other  special  populations  should  be  studied.  At  a  minimum,  the  research 
efforts  of  the  two  organizations  should  be  coordinated  to  avoid  redundancy. 

How  can  the  Army  improve  recruiting  success  in  locations  with  traditionally  low 
propensity  to  serve  (e.g.,  in  urban  areas)?  This  was  an  issue  raised  in  several  of  our  interviews 
with  RC  personnel,  particularly  the  ARNG.  It  may  be  that  NRS  data  could  be  used  to  explore 
this  issue  at  a  fairly  general  level.  A  more  promising  strategy,  however,  would  be  to  try  to 
identify  natural  experiments  from  which  to  draw  conclusions  about  the  effectiveness  of  various 
strategies.  This  would  involve  identifying  RC  units  that  are  trying  different  strategies  to  recruit 
in  “hard  to  reach”  areas,  and  attempting  to  match  them  to  comparable  units  not  applying  those 
strategies.  Alternatively,  a  more  qualitative  case  study  approach  should  yield  informative 
insights  into  what  works  and  what  does  not. 

Are  there  effective  ways  in  which  RC  recruiting  can  be  linked  to  AC  recruiting  and 
turnover?  USAREC  is  responsible  for  recruiting  for  the  AC  and  the  USAR,  but  is  not  involved 
in  ARNG  recruiting  activities.  Even  though  AC  and  USAR  recruiting  are  handled  under  the 
same  command,  however,  there  is  little  evidence  that  the  two  activities  are  well-integrated  so  that 
they  maximize  benefits  for  both  components.  And  to  the  extent  that  the  AC  is  downsized 
relative  to  the  RC,  it  is  incumbent  on  the  RC  (both  the  USAR  and  ARNG)  to  effectively  draw 
from  the  decreasing  pool  of  prior  service  individuals. 

Explicit  coordination  and  cooperation  among  the  three  components  on  enlisted  recruiting 
matters  would  be  beneficial  for  all,  perhaps  particularly  the  ARNG  since  there  is  currently  no 
coordination  with  AC  recruiting  (COL  Nelson,  COL  Sullivan,  &  COL  Carpenter,  personal 
communication,  December  17, 1996).  It  is  possible  that  offering  research  support  would  be  a 
means  of  encouraging  such  collaboration.  For  example,  review  of  recruiting  processes  and 
procedures  and  might  identify  points  at  which  an  AC  recruiter  should  consider  steering  a 
candidate  to  the  USAR  or  ARNG.  Similarly,  review  of  separation  procedures  might  identify 
ways  to  increase  coordination  for  RC  service. 

How  can  theArn^  reduce  drop-outs  from  volunteer  deployments?  There  seems  to  be  a 
considerable  amount  of  information  available  to  address  this  problem,  particularly  if  one  includes 
information  drawn  from  deployment  experiences  that  are  not  necessarily  volimtary.  What  may 
be  of  most  use  at  this  point  is  to  consolidate  issues  identified  in  different  studies  and  develop 
plans  for  implementing  solutions.  Much  of  this  work  would  seem  to  be  operational  rather  than 
research  in  nature.  But,  at  a  minimum,  it  would  be  valuable  to  provide  support  by  gathering 
reactions  to  planned  changes  in  procedures  (e.g.,  through  surveys  of  previously  deployed 
personnel)  and  formally  evaluating  future  mobilization  efforts. 
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Retention/T urnover 


The  interviews  we  conducted  with  RC  officers  during  the  course  of  this  project  revealed 
significant  concerns  about  retention  issues.  Although  some  turnover  from  military  service  is  to 
be  expected,  continued  extensive  losses  of  trained  personnel  drains  training  resources  and 
threatens  readiness.  Increased  concerns  about  retention  are  grounded  in  a  number  of  factors. 
These  include  (a)  the  nature  and  number  of  missions  that  RC  personnel  are  being  asked  to 
support,  (b)  the  possibility  that  the  Guard  will  be  losing  combat  positions  through  force 
restructuring,  and  (c)  the  uncertainty  of  downsizing.  The  concern  with  the  first  two  factors  is  that 
reservists  are  saying  “I  didn’t  sign  up  for  this.” 

Turnover  research  has  a  long  history,  producing  a  number  of  models  that  seek  to  explain 
organizational  withdrawal  behaviors  (e.g.,  Mobley,  1977;  Price  &  Mueller,  1981;  Steers  & 
Mowday,  1981).  Most  of  these  models  incorporate  variables  related  to  attitudes  about  the  job 
(usually  job  satisfaction)  and  perceived  alternatives  for  other  jobs.  The  focus  of  these  models  has 
been,  at  least  implicitly,  on  civilian  turnover  from  frill  time  jobs. 

Knapp  (1993)  and  others  have  discussed  the  need  to  distinguish  between  civilian  and 
military  turnover,  and  between  different  types  of  military  turnover.  Military  turnover  in  the  AC 
can  mean  failure  to  fulfill  one’s  contractual  obligation  (attrition)  or  it  can  mean  the  failure  to 
reenlist.  In  the  active  military,  the  former  problem  is  typically  of  greater  concern  than  the  latter. 
The  need  for  distinguishing  between  active  and  reserve  turnover  also  seems  apparent  (Horn, 
Katerberg,  &  Hulin,  1979).  RC  personnel  are  primarily  part  time  rather  than  fiill  time,  and  so  the 
factors  associated  with  the  turnover  decision  may  be  different.  Despite  these  conceptual 
distinctions,  meta-analyses  of  the  turnover  literature  have  typically  combined  results  drawn  from 
military  and  civilian  studies  (e.g..  Steel  &  Ovalle,  1984).  If  only  because  the  turnover  base  rate 
is  typically  much  higher  for  military  nonreenlistment  than  for  civilian  turnover,  direct 
comparisons  of  point-biserial  correlations  are  not  meaningful.  Indeed,  at  least  one  study  used 
structural  equation  modeling  to  show  that  whether  data  were  gathered  in  military  or  civilian 
settings  served  as  a  moderator  in  the  prediction  of  turnover  (Horn,  Caranikas-Walker,  Prussia,  & 
Griffeth,  1992).  Thus,  while  civilian-based  turnover  research  literature  can  and  should  be  used  to 
help  in  the  study  of  military  attrition  and  reenlistment  behavior,  it  is  not  sufficient  for 
understanding  either  the  AC  or  RC  military  turnover  environment. 

AC  Turnover  Research 

It  has  long  been  known  that  first-term  attrition  for  enlisted  personnel  is  most  accurately 
predicted  by  educational  status  (e.g..  Flyer,  1959,  Laurence,  1993;  McCloy,  DiFazio,  &  Carter, 
1993).  This  is  true  for  both  AC  and  RC  personnel,  and  it  is  why  high  school  diploma  status  is  an 
important  characteristic  established  in  recruiting  quotas.  It  is  also  a  common  finding  that  women 
are  more  likely  to  attrit  than  men. 

LaRocco,  Pugh  and  Gunderson  (1977),  Royle  and  Robertson  (1980),  Youngblood, 
Mobley,  and  Meglino  (1983),  Motowidlo  and  Lawton  (1984),  Farkas  and  Tetrick  (1989),  Knapp, 
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McCloy,  and  DiFazio  (1993),  and  Steel  (1996)  have  studied  reenlistment  behavior  of  active  duty 
personnel  primarily  from  a  psychological  perspective.  Most  of  these  studies  have  used  job 
satisfaction  as  a  primary  predictor  of  turnover  and  have  shown  that  job  satisfaction  is  positively 
associated  with  the  intention  to  reenlist  and  actual  reenlistment  behavior. 

RC  Turnover  Research 

There  are  a  number  of  studies,  conducted  by  ARI  and  others,  of  turnover  behavior  in  the 
RC  that  have  taken  an  econometric  perspective  (e.g..  Dale,  1987, 1989;  Grissmer,  Burright, 
Doering,  &  Sachar,  1982).  Most  of  this  research  is  based  on  the  moonlighting  theory,  developed 
by  Shishko  and  Rosteker  (1976),  which  posits  that  the  decision  to  participate  in  a  part  time  job  is 
driven  by  an  evaluation  of  the  net  monetary  and  nonmonetary  benefits  of  doing  so.  The  amount 
of  variance  in  turnover  typically  accoimted  for  in  this  research  is  often  disappointingly  low. 

Some  of  this  research  has  focused  on  the  effects  of  a  single  economic  motivator;  reenlistment 
bonuses.  For  example,  Rand  conducted  a  series  of  studies  to  examine  the  effects  of  bonuses  on 
reenlistment  behavior  across  all  military  reserve  components  (e.g.,  Grissmer  et  al.,  1982).  The 
1978  reenlistment  bonus  test  showed  that  bonuses  did  not  greatly  increase  the  number  of  persons 
who  reenlisted,  but  they  did  help  extend  reenlistment  terms  and  reduced  subsequent  attrition. 

Although  there  are  several  psychologically-oriented  models  that  have  been  proposed  to 
explain  turnover  behavior,  Horn  and  his  colleagues  selected  a  more  general  behavior  prediction 
model  in  an  attempt  to  explain  turnover  from  the  ARNG  (Horn  et  al.,  1979;  Horn  &  Hulin,  1981; 
Miller,  Katerberg,  &  Hulin,  1979).  Fishbein  and  Ajzen  (1975)  theorized  that  behavior  is  a 
function  of  a  person’s  attitude  toward  the  behavior  and  how  significant  others  regard  the 
behavior.  The  research  that  shows  that  RC  reenlistment  behavior  is  heavily  influenced  by  the 
views  of  employers  and  spouses  suggests  that  this  type  of  model  is  well-suited  to  the  RC 
environment.  Horn  et  al.  collected  data  on  ARNG  reenlistment  in  the  late  1970's  and  found 
general  support  for  this  model. 

It  seems  clear  that  the  most  effective  strategy  for  predicting  reserve  reenlistment  will  be 
to  include  both  psychological  (e.g.,  patriotism,  job  satisfaction)  and  economic  variables  relevant 
to  this  environment.  For  example,  both  Grissmer,  Kirby,  and  Sze  (1992)  and  Lakhani  and  Fugita 
(1993)  analyzed  data  from  the  1986  Reserve  Components  Survey  of  Enlisted  Persoimel  to  study 
reenlistment  behavior.  Grissmer  et  al.  looked  at  service-related  variables  (e.g.,  component,  term 
of  enlistment),  demographic  characteristics  (e.g.,  gender,  age),  and  civilian  job  characteristics 
(e.g.,  net  wage,  supervisor’s  attitude  toward  service).  In  this  sample  of  over  5,000  RC  personnel, 
the  lowest  retention  rates  were  in  the  National  Guard  and  Marine  Corps  reserve.  Factors 
associated  with  failure  to  reenlist  included  perceived  supervisor  and  spouse  attitudes  toward  the 
reserve;  dissatisfaction  with  training,  equipment,  and  morale  of  the  unit;  and  lower  paygrades. 
Lakhani  and  Fugita  compared  the  moonlighting  and  a  more  psychologically-oriented 
“patriotism”  models  of  reenlistment  using  data  from  the  1986  RC  Survey.  They  found  that 
spouse’s  attitude  was  an  even  stronger  predictor  of  reenlistment  than  reserve  earnings. 
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Examination  of  the  reenlistment  intentions  of  USAR  reservists  with  less  than  30  months 
in  service  showed  that  difficulties  getting  benefits  (essentially  paperwork  problems)  was  a 
significant  problem  and  that  this  was  a  factor  in  their  intentions  to  reenlist  (Amerind,  1996b). 
Other  issues  included  pay  and  quality  and  amount  of  training.  The  same  researchers  studied 
educational  support  specialists  who  complained  of  poor  computer  support,  understaffing,  and  other 
problems  that  resulted  in  relatively  poor  support  of  USAR  personnel  seeking  access  to  their 
educational  benefits. 

Steel  (1996)  examined  the  impact  of  alternative  job  availability  (both  real  and  perceived)  on 
the  reenlistment  behaviors  of  Air  Force  RC  personnel.  Using  both  the  objective  and  perceptual 
indices  of  job  alternatives  together  provided  greater  understanding  of  the  reenlistment  decision  than 
either  used  alone. 

There  have  been  other  research  studies  that  have  examined  specific  elements  of  the 
reenlistment  issue  and/or  special  populations.  For  example,  Amerind  (1996a)  found  that  USAR 
physicians  planning  to  leave  before  retirement  most  fiequently  cited  (in  order  of  fi^uency  of 
mention)  the  financial  impact  of  mobilization,  civilian  employment  demands,  family  demands, 
potential  for  future  mobilizations,  and  low  pay  as  the  primary  reasons  for  wanting  to  leave.  Those 
physicians  wiio  indicated  that  they  intended  to  stay  cited  service  to  country,  retirement  benefits,  and 
their  overall  Reserve  experience  as  their  primary  reasons  for  wanting  to  stay.  The  sampled  physicians 
indicated  great  interest  in  a  Ready  Reserve  Mobilization  Income  Insurance  program. 

As  discussed  in  more  detail  in  Volume  1,  Ramsberger  and  DiFazio  (1994)  found  that 
Reserve/Guard  registered  nurses  were  concerned  about  downsizing,  feelings  about  the  Army,  job 
satisfaction,  and  the  possibility  of  serving  in  combat  when  considering  their  career  intentions.  In  an 
earlier  study  conducted  for  USAREC,  a  survey  of  USAR  nurses  showed  dissatisfaction  with 
communication,  paperwork  inefficiency,  annual  training  inflexibility,  and  limited  training  (Miller  & 
Ashton,  1 990).  However,  for  the  respondents  in  this  study,  benefits  rather  than  leadership  or  other 
factors  were  the  most  important  reason  for  remaining  in  the  RC. 

Research  Questions 

As  a  general  comment,  the  turnover  literature  is  plagued  with  studies  that  have  failed  to  use 
the  most  appropriate  multivariate  statistical  methods  to  analyze  their  data.  For  most  civilian  turnover, 
for  example,  event  history  analysis  is  the  preferred  method,  though  psychologists  are  only  recently 
beginning  to  leam  about  and  use  it  (Dickter,  Roznowski,  &  Harrison,  1996;  McCloy,  1993).  How 
researchers  handle  turnover  base  rate  information  is  also  an  important  aspect  of  the  analysis  and 
interpretation  of  turnover  data.  Thus,  it  is  important  that  any  research  examining  RC  turnover 
behavior  be  based  on  careful  consideration  of  alternative  analytic  methods. 

What factors  are  most  highly  related  to  retention,  particularly for  understaffed  positions? 
There  has  been  a  great  deal  of  work  here  that  at  this  point  needs  to  be  integrated  into  a  comprehensive 
RC-specific  model  of  turnover  behavior.  Prior  research  has  shown  that  this  model  should  include 
variables  such  as  spouse  and  employer  support,  job  satisfaction,  unit  morale,  satisfaction  with 
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equipment  and  training,  leadership  factors,  camaraderie,  and  administrative  support  issues.  Of 
particular  interest  to  RC  persormel,  based  on  our  interviews,  is  a  better  understanding  of  how 
leadership  and  training  influence  retention.  The  consensus  of  opinion  from  those  we  spoke  with  is 
that  intense  training  and  utilization  are  good  for  retention.  But  there  is  also  concern  that  putting  too 
many  requirements  on  members  may  make  retention  more  of  a  problem.  One  officer,  for  example, 
recommended  research  to  identify  the  “too  much”  line  (COL  D.  Youngman,  personal 
communication,  December  12, 1996).  A  mote  comprehensive  model  would  clarify  the  nature  and 
magnitude  of  the  roles  of  leadership  and  training  on  retention. 

What  strategies  would  help  reduce  turnover?  Several  researchers  have  already  pointed  out 
the  need  to  implement  policies  to  appeal  to  the  concerns  of  employers  and  spouses  in  order  to  increase 
reenlistment  (Grissmer  et  al.,  1982;  Lakhani  &  Fugita,  1993).  The  research  clearly  indicates  their 
importance  to  the  reenlistment  decision,  so  the  question  now  becomes  to  what  extent  can  the  Army 
influence  their  opinions.  We  would  also  suggest  a  more  expansive  ^proach  initially,  systematically 
identifying  those  factors  influencing  the  retention  decision  over  which  the  Army,  through  its  policies 
and  procedures,  can  have  at  least  some  influence.  Administrative  problems  (e.g.,  poor  record-keeping, 
training  scheduling  difficulties,  delays  in  obtaining  pay  and  benefits)  cited  in  several  pieces  of 
research,  for  example,  are  certainly  under  the  control  of  the  Army.  Resource  constraints  may  lead  to 
inattention  to  such  details,  but  research  that  clearly  demonstrates  their  impact  on  turnover  may  provide 
the  necessary  incentive  to  make  such  issues  a  higher  operational  priority. 

What  are  the  effects  of providing  realistic  job  information  to  prospects?  There  is  some 
evidence  that  reservists  are  dis^pointed  with  their  RC  experience,  which  might  make  them  more 
inclined  to  leave  the  service.  Though  providing  prospects  with  realistic  expectations  (e.g.,  the  RC  is 
likely  to  take  more  time  than  it  says  on  paper)  may  hurt  recruiting  efforts,  the  payoff  could  be  in 
increased  retention.  It  would  seem  worthwhile  to  explore  the  possibilities  further  by  devising  ways 
for  recruiters  to  convey  such  information  effectively  and  to  evaluate  the  utility  of  doing  so. 

How  can  turnover  be  defined  and  tracked  to  facilitate  turnover-related  research  and 
management?  A  perpetual  problem  in  the  study  of  turnover  is  the  difficulty  of  measuring  this 
seemingly  objective  variable.  A  typical  problem  is  having  involuntary  turnover  intentionally 
categorized  as  voluntary  turnover  (e.g.,  a  poor  performer  agrees  to  leave  when  asked  to  do  so). 
Turnover  from  the  RC  is  complicated  enough  to  suggest  that  some  attention  be  paid  to  how 
different  types  of  turnover  are  grouped  for  purposes  of  turnover  research  and  management,  and 
then  assuring  that  persormel  files  maintain  the  required  information  accurately.  There  are  many 
distinctions  that  might  be  made:  officer  versus  enlisted  attrition,  continuation  (staying  from  year 
to  year),  and  reenlistment;  leaving  the  service  altogether  or  leaving  for  the  IRR  or  another 
component;  leaving  early  in  one’s  career  versus  leaving  later;  separation  of  part  time  versus  full 
time  persormel.  Greater  attention  to  how  researchers  treat  these  different  variations  of  RC 
“turnover”  is  bound  to  increase  our  ability  to  understand  it. 
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Selection,  Classification,  and  Performance  Management 

We  found  little  evidence  of  RC-specific  work  conducted  for  the  Army  or  any  of  the  other 
services  related  to  this  research  domain.  A  small  exception  was  a  study  performed  at  the  Naval 
Postgraduate  School  for  USAREC  on  the  selection  of  active  Guard  and  Research  recruiters 
(G.  W.  Thomas,  1987).  Presumably  this  relative  lack  of  research  emphasis  is  because  (a)  related 
research  conducted  in  the  AC  is  typically  generalizable  to  the  RC  and/or  (b)  the  way  that  the  RC 
has  traditionally  operated  has  made  such  research  a  relatively  low  priority,  particularly  compared 
to  research  needs  in  the  training  arena.  But  trends  affecting  the  RC  suggest  some  work  in  this 
area  that  would  help  the  RC  use  its  human  resources  more  effectively  in  its  future  environment. 

This  is  a  relatively  eclectic  discussion  section,  in  that  we  zero  in  on  a  diverse  set  of  needs 
that  were  identified  in  this  project  as  warranting  research  attention.  Specifically,  needs  in  four 
areas  are  reviewed:  (a)  selecting  enlisted  personnel  with  technical  skills,  (b)  supporting 
reclassification  requirements,  (c)  developing  individual  performance  measures  and  standards,  and 
(d)  documenting  unit  administrator  task  requirements. 

In  addition  to  the  topics  discussed  here,  the  reader  is  referred  to  a  Joint-Service  selection 
and  classification  research  roadmap  presented  by  J.P.  Campbell  et  al.  (1994).  That  report 
discusses  critical  research  questions  in  the  following  areas:  (a)  building  a  Joint-Service  policy 
and  forecasting  database,  (b)  developing  new  job  analysis  methodologies,  (c)  capturing  criterion 
(job  performance  measurement)  policy,  (d)  conducting  criterion  measurement  research,  (e) 
conducting  predictor-related  research,  (f)  modeling  classification  decisions,  and  (g)  investigating 
test  fairness  issues.  Although  some  of  the  research  questions  are  more  relevant  to  the  AC  than  to 
the  RC,  this  research  agenda  was  developed  for  the  total  Joint-Service  force. 

Selecting  Enlisted  Personnel  with  Technical  Skills 

Force  downsizing  and  restructuring  activities  can  have  significant  implications  for  the 
ways  in  which  personnel  are  selected,  classified,  and  utilized.  One  scenario  raised  in  several  of 
the  project  interviews  was  the  possibility  of  selecting  for  certain  skills  rather  than  training  for 
those  skills.  This  strategy,  if  proven  feasible,  would  allow  increasingly  limited  Army  training 
resources  to  be  focused  on  skills  specific  to  the  Army. 

The  feasibility  of  this  strategy  rests  most  fundamentally  on  the  availability  of  applicable 
training  outside  the  military.  The  military  is  already  accustomed  to  accepting  the  education  and 
civilian  credentialing  of  physicians,  nurses,  engineers,  and  selected  other  professional  skills 
required  of  its  officers.  And  in  the  USAR,  in  which  most  enlisted  positions  are  combat  service 
support,  there  are  many  required  job  skills  that  can  be  obtained  outside  the  military  (e.g.,  truck 
driving,  medical  support).  Various  civilian  credentialing  programs  document  competence  in 
some  of  these  skills,  but  not  in  others.  In  the  ARNG  as  it  is  currently  structured,  most  positions 
are  combat  and  combat  support,  making  this  option,  for  the  time  being  at  least,  not  as  feasible. 
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The  idea  of  recruiting  enlisted  personnel  with  the  requisite  technical  job  skills  and 
bringing  them  in  at  upper  grade  levels  (E5  or  higher)  was  raised  in  interviews  with  both  USAR 
and  ARNG  personnel.  They  were  interested  in  the  potential  for  saving  precious  training 
resources  and  reducing  skill  decay  by  accessing  people  who  perform  similar  work  in  their 
civilian  jobs.  Nevertheless,  there  will  likely  be  institutional  resistance  to  this  type  of  change  to 
the  persoimel  management  system.  Technical  training,  both  for  initial  MOS  classification  and 
for  reclassification  requirements  (see  below),  is  currently  governed  by  a  relatively  lock-step  Total 
Army  School  system.  Speeding  up  training  and/or  reducing  training  requirements  would  be  a 
mixed  blessing  for  this  system. 

Research  Questions 

For  which  MOS  could  technical  skills  be  selected  for  rather  than  trained?  Although 
there  may  be  several  other  factors  that  determine  the  feasibility  of  selecting  for  certain  skills 
rather  than  training  them,  perhaps  the  most  fimdamental  question  is  which  MOS  have  enough 
overlap  with  civilian  skills  to  make  this  an  option  worth  considering  further?  One  strategy  would 
be  to  identify  possible  MOS  at  a  fairly  global  level  (e.g.,  based  on  job  titles).  A  crosswalk  of 
military  to  civilian  occupations  developed  for  DoD  (Wright,  1984)  would  be  helpful  for  this  level 
of  inquiry.  Of  course,  most  combat  positions  would  have  to  be  eliminated  early  on,  as  they  are 
unlikely  to  have  close  civilian  counterparts.  The  next  step  would  be  to  use  more  detailed 
occupational  analysis  information  to  identify  the  nature  and  number  of  skills  required  by  an 
MOS  that  appear  to  be  covered  in  civilian  education,  training,  or  job  opportunities. 

For  selected  MOS,  how  could  knowledge  and  skill  acquisition  be  confirmed  prior  to 
selection?  There  are  several  options  that  might  be  available  to  the  Army  in  determining  whether 
an  individual  has  sufficient  knowledge  and  skill  to  do  the  technical  work  required  in  some  MOS. 
The  easiest  strategy  would  be  to  identify  civilian  educational,  licensure,  and/or  certification 
credentials  that  would  provide  sufficient  evidence  of  knowledge/skill  acquisition.  In  this  way, 
the  Army  could  take  advantage  of  the  ever  increasing  numbers  of  certification  programs  for 
“nonprofessional”  occupations  that  are  available  in  the  civilian  sector.  Another  strategy  would 
be  to  accredit  civilian  training  programs  to  assure  that  they  meet  the  Army’s  needs,  then  accept 
successful  completion  of  such  programs  as  evidence  of  skill  acquisition.  If  these  strategies  are 
not  sufficient,  tests  could  be  developed  that  individuals  would  have  to  pass  before  being  selected 
for  certain  MOS.  Such  certification  tests  would  be  the  most  direct  demonstration  of  the  nature 
and  level  of  competence  required  by  the  Army.  They  would  also  have  potential  as  recruiting  and 
counseling  tools. 

Supporting  Reclassification 

In  the  AC,  classification  of  enlisted  personnel  is  a  critical  decision  point  for  which  a  great 
deal  of  research  has  been  performed  (e.g.,  J.P.  Campbell  &  Zook,  1995).  In  the  RC,  however, 
classification  is  handled  much  differently.  To  the  extent  possible,  classification  is  based  on  an 
individual’s  prior  service  MOS  because  this  minimizes  training  requirements  to  reach  MOSQ.  In 
the  ARNG,  designation  of  the  unit  to  which  an  individual  is  joining  (e.g.,  infantry,  armor)  will 
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largely  determine  MOS  assignment.  Indeed,  a  relatively  common  occurrence  is  for  a  unit’s 
designation  to  change,  resulting  in  the  need  to  reclassify  and  therefore  retrain  individual 
reservists. 

Whatever  the  specific  method  for  classification  or  reclassification,  explicit  consideration 
of  an  individual’s  aptitude  and  qualifications  is  not  a  significant  part  of  the  RC  equation  as  it  is 
with  the  AC.  But  ARI  research  could  support  the  RC  in  the  reclassification  arena  by  looking  for 
any  “MOS-specific”  job  requirements  that  overlap  across  selected  MOS.  The  idea  is  that 
commonalties  across  MOS  (e.g.,  between  1  IB-infantry  and  13B-cannon  crewmember)  could  be 
used  to  reduce  the  retraining  requirement  when  reservists  are  reclassified. 

Research  Questions 

For  selected  MOS,  what,  if  any,  technical  skill  requirements  overlap?  The  first  step 
here  would  be  to  identify  MOS  which  have  the  highest  likelihood  of  being  affected  by  large- 
scale  reclassification  (e.g.,  of  several  units)  and  select  a  small  number  (two  to  three)  to  test  this 
concept.  The  next  step  would  be  to  examine  information  about  job  requirements  to  identify 
what,  if  any,  redimdancies  exist.  Common  soldiering  tasks  would  be  easy  to  identify  as  they  are 
identified  through  doctrine.  The  search  for  other  commonalties  will  be  complicated  by  the 
probability  that  there  will  be  little  overlap  in  tasks  (other  than  those  that  are  common  across  all 
MOS).  Rather,  it  is  more  likely  that  vinderlying  skills  (e.g.,  troubleshooting,  record-keeping)  will 
show  redundancy  that  could  then  be  exploited  through  modified  training  programs.  It  is 
relatively  easy  to  obtain  lists  of  MOS  tasks,  but  the  knowledge  and  skill  requirements  underlying 
the  ability  to  perform  those  tasks  may  need  to  be  identified  before  this  research  goal  could  be 
achieved. 

What  savings  are  achieved  through  reduced  reclassification  training?  If  there  are  any 
training  resources  saved  as  a  result  of  systematic  efforts  to  identify  knowledge/skill  redundancy 
across  various  MOS,  the  next  question  will  be  whether  the  savings  offset  the  costs  of  identifying 
these  redimdancies.  This  will  require  an  assessment  of  the  costs  associated  with  the  identified 
methodology  and  the  savings  achieved  through  redesigned,  presumably  more  abbreviated, 
retraining  activities. 

Individual  Performance  Measures  and  Standards 

Group  level  performance  measurement  was  discussed  previously  in  the  context  of 
training.  In  this  section,  we  want  to  raise  the  importance  of  individual  performance 
measurement.  Assessment  at  the  individual  level  can  serve  a  variety  of  purposes,  including  (a) 
criterion  measurement  in  selection  and  classification  research,  (b)  measurement  of  individual 
skill  acquisition  to  determine  MOSQ  and/or  provide  developmental  feedback,  and  (c)  support  for 
force  integration  planning  and  research. 

ARI  has  conducted  a  great  deal  of  research  in  the  area  of  individual  performance 
measurement,  primarily  for  its  use  as  a  criterion  measure.  Indeed,  through  the  Project  A  research 
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program,  the  Army  has  led  academia  and  industry  in  making  scientific  and  practical  strides  in 
this  area.  Though  mostly  based  on  work  with  AC  personnel,  this  research  is  generalizable  to  the 
RC.  Standardized  hands-on  and  written  tests  of  task  knowledge  and  skill  and  rating  instruments 
suitable  for  peers  and  supervisors  have  been  developed  for  the  MOS  listed  in  Table  8.  These 
measures  cover  both  MOS-specific  and  common  soldiering  skills.  Though  they  may  require 
updating  and/or  tailoring,  these  measures,  and  the  procedures  for  developing  them,  are  available 
to  the  RC  for  test  validation  or  training  and  development  needs. 

The  Project  A  work  successfiilly  demonstrated  that  performance  on  the  Armed  Services 
Vocational  Aptitude  Battery  (ASVAB)  and  other  experimental  selection  and  classification 
measures  was  correlated  with  performance  on  a  variety  of  job  performance  measures.  This  work 
helped  illustrate  that  performance  is  multidimensional  and  that  there  are  a  variety  of  ways  to 
assess  various  elements  of  performance  (e.g.,  technical  skill  proficiency,  leadership),  none  of 
which  is  flawless.  Rather,  a  “whole-person”  view  of  performance  is  best  achieved  through  the 
use  of  multiple  measurement  methods  to  assess  success  in  both  “can-do”  and  “will-do”  aspects  of 
performance. 


Table  8. 


MOS  with  Pro 

ect  A-Developed  Job  Performance  Measures. 

IIB 

Infantryman 

13B 

Caimon  Crewmember 

19E/K 

Tank  Crewmember 

31C 

Single  Chaimel  Radio  Operator 

63B 

Light  Wheeled  Vehicle  Mechanic 

71L 

Administrative  Clerk 

88M 

Truck  Driver 

91A/B 

Medical  Specialist 

95B 

Military  Police 

But  Project  A  researchers  did  not  have  to  establish  performance  standards  on  either  the 
new  predictor  tests  or  the  job  performance  tests  that  they  developed.  Training  programs,  on  the 
other  hand,  must  set  performance  standards  to  assure  that  trainees  have  reached  MOSQ  before 
they  are  allowed  to  leave  training.  And  if  efforts  proceed  to  enlist  individuals  who  have 
demonstrated  sufficient  skill  in  particular  areas,  standards  vvill  have  to  be  set  for  the  tests  used  to 
certify  those  skills.  Force  integration  policy  and  research  is  another  area  in  which  the  issue  of 
performance  standards  must  be  addressed.  Force  integration  was  a  topic  that  frequently  surfaced 
during  om  project  interviews,  and  it  is  one  that  is  fraught  with  sensitivity  because  it  raises  the 
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question  of  whether  RC  personnel  are  trained  well  enough  to  work  effectively  with  AC 
personnel. 

Research  Questions 

What  types  of  individual  perfornuince  standards  do  Army  researchers,  trainers,  and 
policy-makers  need?  This  is  clearly  a  question  that  should  be  posed  with  an  eye  toward  both  AC 
and  RC  personnel  simultaneously,  and  a  good  example  is  MOSQ.  Interviews  with  RC  leaders 
showed  a  strong  desire  for  alternatives  to  course  completion  to  determine  whether  a  soldier  in  a 
given  MOS  is  qualified  to  serve  in  that  MOS.  For  some  purposes,  it  may  be  sensible  to  have 
more  than  one  standard.  One  level  of  performance  might  be  acceptable  for  training  environments 
whereas  another  higher  level  might  be  required  for  more  demanding  environments  or  for  higher 
ranking  soldiers.  It  is  conceivable,  but  probably  not  politically  palatable,  to  have  different 
minimum  standards  for  RC  and  AC  personnel,  given  the  higher  expectations  of  soldiers  who  are 
training  fiill-time  rather  than  part-time.  In  any  case,  the  point  here  is  to  do  some  kind  of  needs 
analysis  to  determine  where  systematic  procedures  for  establishing  standards  are  required  or 
would  improve  upon  current  strategies. 

What  standard  setting  methodologies  would  be  most  useful for  helping  to  set  Army 
individual  performance  standards?  There  are  strategies  for  establishing  performance  standards 
for  selection  and  certification  tests  that  could  be  adapted  to  the  determination  of  job  performance 
standards  (Angofif,  1971;  Livingston  &  Zieky,  1982).  The  so-called  criterion-referenced 
strategies  generally  involve  the  collection  of  expert  judgments  from  a  sample  of  carefully 
selected  judges.  There  are  more  empirical  strategies  that  are  not  often  feasible  to  apply,  but 
which  might  be  quite  appropriate  for  the  military  environment. 

Documenting  Unit  Administrator  Task  Requirements 

A  specific  issue  raised  by  some  interviewees  was  the  amoimt  of  undocumented  work  that 
unit  administrative  persormel  put  in  to  make  an  RC  unit  work.  The  concern  is  that  units  cannot 
be  successful  unless  unit  administrators  are  willing  to  work  longer  hours  than  prescribed,  and 
that  this  problem  will  only  get  worse  as  downsizing  takes  its  toll  on  the  availability  and 
responsibilities  of  administrative  personnel.  RC  personnel  we  spoke  with  wanted  documentation 
of  the  broad  responsibilities  of  this  key  position  to  help  defend  against  efforts  to  reduce  their 
administrative  resources.  It  may  be,  however,  that  this  question  should  be  framed  more  broadly, 
at  least  initially.  That  is,  it  may  be  that  there  are  other  positions  or  certain  MOS  for  which  RC- 
specific  job  description  information  would  be  useful. 

Research  Questions 

What  positions  require  RC-specific  analysis?  Inquiry  regarding  existing  task  analysis 
information  and  targeted  questions  to  the  RC  about  any  other  positions  would  help  identify 
positions  other  than  unit  administrator  that  merit  analysis.  The  goal  would  be  to  look  for 
positions  that  have  no  analog  in  the  AC  or  for  which  unique  RC  requirements  may  make  separate 
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RC  analysis  necessary.  In  some  cases,  it  may  be  that  the  occupational  analysis  researchers 
already  have  required  data  that  just  need  to  be  packaged  for  use  by  users  outside  of  TRADOC. 
Note  that  the  Air  Force  has  conducted  some  research  looking  at  occupational  analysis  strategies 
across  AC,  RC,  and  civilian  positions  that  might  offer  some  insight  in  this  area. 

What  are  the  responsibilities  of  an  RC  unit  administrator?  This  could  be  a  fairly 
traditional  task  analysis  that  identifies  tasks  performed  in  this  position  and  collects  data  on  task 
importance  and  time  spent  using  a  survey-based  methodology. 

Management  Systems 

This  section  briefly  discusses  issues  of  a  more  operational  nature  than  those  discussed  in 
other  sections.  These  are  issues  that  will  likely  involve  the  use  of  technology  (i.e.,  automation)  or 
organizational  reengineering  to  address  and,  although  ARI  is  not  a  software  developer  or 
organizational  change  consultant,  it  can  provide  design  and  evaluation  research  services  that  would 
assist  in  these  areas. 

Many  of  the  problems  identified  in  ARI  and  other  surveys  of  RC  personnel  attitudes  related 
to  training  and  deployments  are  founded  in  administrative  problems.  Lack  of  communication,  poor 
record-keeping,  limited  access  to  equipment,  and  generally  poor  orchestration  of  large-scale 
activities  (most  notably  the  ODS  call-up)  were  all  cited  as  significant  problems  in  multiple  surveys. 
And  it  is  these  types  of  problems  that  have  been  shown  to  decrease  retention  rates,  a  problem  that 
may  increase  as  more  technologically  savvy  recruits  quickly  become  fiaistrated  with  administrative 
and  training  problems  that  they  consider  easily  surmountable  with  automation. 

Several  RC  persormel  we  interviewed  were  also  concerned  about  the  inefficiencies  and 
other  problems  associated  with  the  decentrali2Kd  way  in  which  assignments  and  promotions  are 
made  in  the  RC.  For  example,  it  is  believed  that  well-trained  personnel  are  imnecessarily  lost  to  the 
RC  when  they  relocate  because  there  is  no  formal  mechanism  for  trying  to  place  the  relocated 
individual  into  a  unit  near  his  or  her  new  home.  There  is  also  considerable  concern  that  personnel 
“game”  the  system,  moving  fi-om  one  unit  to  another  to  get  promotions.  With  more  centralized 
information  systems,  it  might  be  possible  to  make  assignments  and  promotions  in  ways  that  benefit 
the  needs  of  the  RC,  RC  units,  and  individual  soldiers  and  officers.  Such  systems,  assuming  that 
they  can  be  practically  applied,  would  increase  the  fairness  of  promotions  and  assignments,  reduce 
the  need  for  retraining,  and  increase  readiness  by  more  effective  person-position  matching. 

In  an  effort  to  address  many  of  the  concerns  that  have  been  raised  by  USAR  personnel, 
the  Army  Reserve  Personnel  Command  (ARPERSCOM)  is  involved  in  a  major  reorganization. 
Personnel  from  ARPERSCOM  who  were  interviewed  in  this  project  expressed  interest  in  an 
evaluation  of  the  effectiveness  of  their  reform  efforts.  Such  an  evaluation  may  be  within  the 
scope  of  ARI’s  research  mission. 
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Research  Questions 

How  can  training  and  deployment  activities  be  more  effectively  orchestrated?  What 
might  be  useful  here  is  to  consolidate  the  information  from  previous  research  that  documents  the 
magnitude  and  impact  of  problems  in  these  areas.  Such  documentation  may  be  required  to  help 
acquire  the  resources  necessary  to  address  the  problems.  It  would  also  be  useful  to  look  for 
related  problems  that  could  be  addressed  with  common  solutions  (e.g.,  a  more  manageable,  up- 
to-date  personnel  database  accessible  to  all  with  a  need  to  know).  A  less  expansive  strategy 
would  be  to  cull  from  existing  work  the  information  needed  to  identify  and  address  problems  in 
specific  areas  (e.g.,  pay,  access  to  educational  benefits).  An  example  of  this  more  focused 
strategy  is  research  to  improve  the  compatibility  between  the  AC  SIDPERS  and  the  RC 
SIDPERS.  The  systems  will  not  be  fully  compatible  imtil  DoD  adopts  a  standard  personnel 
system.  (Kentucky,  for  example,  is  scheduled  to  be  on-line  in  2001 .)  In  the  interim,  research  to 
develop  bridge  procedures  would  be  welcome  (COL  Smith,  personal  communication,  December 
17, 1996). 

Can  some  aspects  of  assignments  and  promotions  be  effectively  centralized?  The  key 
here  seems  to  be  identifying  a  strategy  that  would  solve  more  problems  than  it  creates.  So  it  would 
make  sense  to  learn  more  about  the  current  system.  How  does  it  work  in  a  “typical”  unit.  Are  there 
formal  or  informal  links  between  assignment  and  promotion  decisions  across  units  in  the  same  or 
different  geographical  areas?  What  works  and  what  does  not  with  the  current  system?  As  the 
current  system  is  better  understood,  it  might  make  sense  to  start  small,  introducing  small  changes  to 
increase  communication  about  unit  needs  and  available  personnel.  More  formal  requirements  to 
centralize  (at  some  level  above  the  unit)  assigiunents  and  promotions,  if  they  show  promise,  could 
be  introduced  perhaps  on  a  pilot  basis  in  one  region  or  another  and  evaluated  before  larger  scale 
changes  are  made  across  the  component  (USAR  or  ARNG). 

Are  ARPERSCOM  efforts  to  reorganize  improving  organizational  mission  effectiveness? 

As  with  any  evaluation  study,  it  will  be  critical  to  identify  what  (e.g.,  pay  delivery,  timeliness  of 
paperwork)  is  expected  to  improve  with  the  reorganized  ARPERSCOM  and  to  identify  or  develop 
reliable  and  valid  measures  of  those  variables.  It  is  also  important  to  establish  baseline  measures  for 
these  variables  prior  to  implementation  of  the  reorganization  effort.  Accordingly,  if  the  reorganization 
is  going  to  be  evaluated,  the  process  should  start  as  soon  as  possible  to  allow  for  this.  It  is  very 
possible  that  available  survey  data  will  be  one  important  source  of  such  baseline  information. 

Leadership 

The  central  role  leadership  plays  in  the  effective  functioning  of  the  military  has  long  been 
understood.  The  Army  has  sponsored  a  significant  amount  of  research  in  the  study  of  leadership, 
and  offers  formal  training  for  leaders  (e.g.,  US  Army  War  College,  CAS^,  US  Sergeants  Major 
Academy)  as  they  step  into  critical  leadership  levels. 

In  this  environment  of  uncertainty  in  the  military,  the  key  role  of  leadership  becomes,  if  it 
is  possible,  even  more  critical.  And  for  the  RC,  which  is  taking  ever  greater  responsibility  and 
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accounting  for  greater  proportions  of  the  Army’s  total  force,  the  days  of  taking  the  leadership 
skills  of  its  officers  and  senior  NCOs  as  a  given  are  over.  The  increasing  demands  of  the  RC  will 
be  most  effectively  met  if  its  leaders  are  given  additional  tools  and  training. 

As  discussed  in  the  critical  questions  listed  below,  we  believe  that  leadership  issues 
should  be  addressed  in  at  least  three  ways:  (a)  identifying  the  characteristics  that  lead  to  success 
as  an  RC  leader,  (b)  developing  training  and  development  tools  that  will  help  RC  leaders 
improve  their  leadership  skills,  and  (c)  looking  at  the  impact  of  leader  attitudes  on  Army  goals, 
most  particularly  in  the  goal  to  develop  a  fully  integrated  total  force. 

We  interviewed  a  TAG  who  estimated  that  the  ARNG  loses  50%  of  its  new  officers 
within  two  years  of  joining  the  RC  (MG  Kane,  personal  communication,  January  12, 1997).  We 
interviewed  other  RC  personnel  and  researchers  who  speculated  that  someone  who  has  been  a 
good  leader  in  the  AC  will  not  necessarily  be  particularly  successful  as  a  leader  in  the  RC.  A  key 
difference  may  be  in  the  need  for  flexibility  demanded  of  an  environment  in  which  schedules, 
equipment,  and  so  forth  carmot  be  as  tightly  controlled  as  they  can  be  in  the  AC. 

The  general  leadership  literature  supports  the  notion  that  there  are  common  characteristics 
of  effective  leaders,  but  that  different  situations  may  demand  somewhat  different  leadership  styles. 
The  Army  has  leadership  doctrine,  as  set  forth  in  Field  Manual  22-100,  that  specifies  leadership 
requirements  in  the  Army  environment.  But  the  question  to  be  asked  is,  are  there  some 
characteristics  that  become  more  or  less  important  to  successful  leadership  in  the  RC  environment? 
The  information  gained  from  this  type  of  research  could  serve  as  the  foundation  for  RC-specific 
leadership  courses  and  development  tools.  It  might  also  eventually  be  used  to  help  identify  those 
ROTC  or  prior  service  officers  with  the  greatest  potential  for  success  in  the  RC. 

ARI,  U.S.  Army  War  College  (USAWC)  and  IDA  researchers  are  working  on  360-degree 
rating  instruments  to  provide  feedback  to  leaders  for  development  purposes  (e.g.,  S.  Halpin, 
personal  communication,  June  19, 1996;  Jacobs,  undated).  The  Strategic  Leader  Development 
Inventory  (SLDI)  is  designed  for  senior  officers  and  is  used  for  students  at  the  USAWC  and  the 
Industrial  College  of  the  Armed  Forces.  The  Azimuth  Check  is  an  adaptation  of  the  SLDI 
developed  by  ARI  to  be  suitable  for  all  officer  ranks.  It  is  currently  being  administered  to 
Captains  attending  the  CAS^  course  at  Ft.  Leavenworth.  Plans  are  to  computerize  this  instrument 
and  make  it  available  for  field  commanders  for  their  own  self-development  activities.  ARI  is 
also  working  on  computerization  of  a  command  climate  survey  that  will  be  made  available  to 
field  commanders  by  the  fall  of  1997.  These  and  other  efforts  can  help  both  AC  and  RC  leaders 
work  on  their  leadership  skills  without  taking  the  time  and  expense  of  attending  off-site  courses. 

Our  interviews  with  RC  personnel  revealed  considerable  frustration  with  the  difficulties 
associated  with  trying  to  be  truly  integrated  with  the  AC  as  envisioned  by  the  total  force  concept. 
Of  particular  concern  is  the  perception  that  AC  leaders,  even  at  the  highest  levels,  do  not  really 
believe  that  the  RC  will  ever  be  able  to  handle  the  role  it  has  been  given,  and  that  this  belief 
trickles  down  through  all  levels,  making  successful  integration  on  a  large  scale  virtually 
impossible.  Clearly,  this  is  a  sensitive  issue.  But  if  these  are  common  perceptions,  then  a  focus 
on  officer  attitudes  is  required  before  real,  long-term  integration  will  be  possible.  The  Army  has 
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faced  this  type  of  situation  with  racial  integration  and  increased  use  of  women  in  the  force.  It 
may  be  that  successful  integration  of  RC  and  AC  personnel  will  require  similar  measures. 

Research  Questions 

What  characteristics  make  leaders  successful  in  the  RC  environment?  There  are  a 
variety  of  ways  that  this  question  might  be  asked,  and  strategies  that  avoid  explicit  measurement 
of  success  of  individual  leaders  will  likely  be  the  most  feasible  to  pursue.  Critical  incident 
analysis,  using  incidents  drawn  from  the  AC  and  the  RC  would  be  one  nonthreatening  way  to 
identify  and  document  differences  both  in  the  types  of  problems  encountered  and  the  behaviors 
required  to  handle  them.  The  OCAR  has  sponsored  research  on  effective  leadership  behavior  in 
the  USAR  (e.g.,  K.  Thomas  &  Barrios-Choplin,  1996)  that  should  also  be  incorporated  into 
future  studies.  In  addition  to  documenting  the  importance  of  leader  behaviors  with  regard  to 
such  areas  as  time  commitment  to  the  unit,  managing  unit  training,  and  improving  standards,  this 
research  has  identified  concrete  best  practices  and  pointed  out  two  common  leadership  problems 
(micromanagement  and  use  of  punishment). 

What  tools  are  useful for  leadership  development  in  the  field?  We  suggest  continued 
support  of  work  related  to  the  SLDI  and  Azimuth  Check,  with  explicit  emphasis  on  special  needs,  if 
any,  of  using  these  instruments  with  RC  personnel.  Some  of  our  interviewees  and  K.  Thomas  and 
Barrios-Choplin  (1996)  also  called  for  increased  mentoring  opportunities  as  a  promising  avenue  for 
leader  development.  Another  suggestion  would  be  to  provide  leaders  with  easily  digestible 
quantities  of  training  and  information  online.  Training-oriented  pieces  would  focus  on  case  studies, 
common  leader  errors,  and  so  forth,  while  information-oriented  pieces  would  focus  on  research 
findings  and  other  information  that  can  help  leaders  perform  their  jobs.  Such  special  topic  reports 
related  to  ARI  research  findings  are  currently  being  developed. 

What  is  the  extent  and  impact  of  component  biases  among  AC  and  RC  leaders?  We 
have  included  this  research  question  because  it  appears  to  be  an  important  one  to  the  success  of 
the  total  force,  not  because  it  would  be  a  welcome  research  topic  in  the  eyes  of  Army  leaders. 
Indeed,  this  may  be  an  area  in  which  quantitative  research  will  be  less  useful  than  qualitative 
research  (e.g.,  case  studies,  interviews,  critical  incident  analyses).  As  with  other  issues  we  have 
mentioned,  the  first  step  is  to  document  the  problem.  The  second  step  is  to  then  incorporate 
content  directed  at  this  issue  into  training  programs,  increase  opportunities  for  collaboration  at  all 
levels  of  the  AC  and  RC,  and  possibly  initiate  programs  ala  diversity  training  exercises. 

OVERALL  AGENDA 

The  critical  research  questions  posed  within  each  research  domain  discussed  in  this  report  are 
listed  in  Table  9.  As  discussed  in  the  introduction,  it  was  difficult  to  obtain  specific  guidance  from 
either  ARI  researchers  or  RC  leaders  regarding  which  research  domains  deserved  the  most  attention, 
let  alone  the  relative  priority  of  individual  research  issues  that  cut  across  those  domains.  What  we 
have  done,  however,  is  attempt  to  capture  overall  impressions  drawn  from  the  researchers  and  leaders 
interviewed  for  this  project  and  merge  these  with  our  knowledge  of  ARI  research  programs  to 
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Table  9. 


Summary  of  Critical  Research  Questions 


Domain 

Area 

Question 

TADSS-Based 

Training 

Cost  Effectiveness 
Trade-Offs 

How  effective  is  a  given  TADSS  in  the  RC 
environment? 

What  savings  associated  with  TADSS-based 
training  should  be  expected? 

Can  performance  categories  be  identified  for 
which  alternative  TADSS  designs  are  most 
effective? 

Part-Task  Training 

How  should  a  task  be  segmented  for  part-task 
training? 

What  is  the  optimum  mix  of  part-  and  whole- 
task  devices? 

TADSS  Fidelity 
Requirements 

How  should  secondary  fidelity  criteria  be 
combined  to  predict  transfer? 

What  measures  should  be  applied  for  each 
fidelity  attribute? 

What  methods  should  be  applied  to  improve  the 
quality  of  secondary  fidelity  criteria? 

How  do  soldiers’  reactions  to  low  fidelity 

TADSS  impact  on  the  true  importance  of 
fidelity? 

How  accurate  are  predictions  by  a  transfer 
prediction  model? 

Training  Support 

What  content  should  be  developed  for  TADSS 
training? 

How  well  do  TSPs  support  implementation  of 
TADSS  training  in  the  RC  environment? 

Computer-Generated 
Forces  (CGF) 

Are  computer-generated  OPFOR  justified? 

How  should  requirements  and  parameters  for 

CGF  be  established? 

How  are  performance  and  training  effectiveness 
affected  by  simplifications  in  the  representation 
of  CGF? 

Micro-Level  Strategies 

How  should  TADSS-based  training  be 
implemented  in  the  RC  environment? 

What  learning  variables  promote  skill 
acquisition,  retention,  and  transfer? 

Performance 

Measurement 

Diagnostic  Measures 

What  level  of  detail  is  needed  for  diagnostic 
instruments? 
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Proficiency  Assessment 

How  should  performance  assessment 
measurement  systems  be  structured? 

What  is  the  reliability  and  validity  of  MOE? 

Training 

Management 

Macro-Level  Training 
Strategies 

What  tasks  should  be  trained  in  TADSS-based 
and  field  events  during  an  RC  unit’s  training 
cycle? 

How  does  the  basis  of  issue  of  TADSS  affect  the 
macro-level  strategy  for  RC  units? 

Database  Development 
and  Maintenance 

What  interventions  or  interruptions  have  an 
impact  on  imit  performance? 

How  can  the  armor/mechanized  infantry 
database  be  maintained  and  accessed? 

How  should  variables  be  aggregated  for  a 
collective  training  database? 

Distance 

Training 

Distance  Learning  (DL) 

What  is  the  most  cost-effective  combination  of 
media  in  DL? 

What  types  of  interactivity  are  required  in  DL? 
What  skills  are  required  for  instructors  to  be 
effective  in  DL? 

What  savings  can  be  anticipated  from  DL? 

Distributed  Interactive 
Simulations  (DIS) 

What  savings  can  be  achieved  by  conducting 
collective  training  in  a  DIS  mode? 

How  can  tactical  engagement  simulation 
systems  be  integrated  to  support  training  in  a 

DIS  mode? 

Additional 

Areas  for 

Training 

Research 

i 

! 

What  are  the  effective  training  delivery 
techniques? 

What  is  the  training  delivery  capability  of  the 
Internet? 

Are  there  alternative  evaluation  models? 

Recruiting 

What  strategies  for  recruiting  enlisted  personnel 
are  most  effective? 

What  strategies  for  recruiting  officers,  including 
those  with  specialized  skills  (e.g.,  physicians, 
engineers)  are  most  effective? 

How  can  the  Army  improve  recruiting  success  in 
locations  with  traditionally  low  propensity  to 
serve  (e.g.,  in  urban  areas)? 

Are  there  effective  ways  in  which  RC  recruiting 
can  be  linked  to  AC  recruiting  and  turnover? 

How  can  the  Army  reduce  drop-outs  from 
volunteer  deployments? 
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Retention/T  umo  ver 

What  factors  are  most  highly  related  to  retention, 
particularly  for  understaffed  positions? 

What  strategies  would  help  reduce  turnover? 

What  are  the  effects  of  providing  realistic  job 
information  to  prospects? 

How  can  turnover  be  defined  and  tracked  to 
facilitate  turnover-related  research  and 
management? 

Selection, 
Classification,  and 
Performance 
Management 

Selecting  Enlisted 
Personnel  with 
Technical  Skills 

For  which  MOS  could  technical  skills  be 
selected  for  rather  than  trained? 

For  selected  MOS,  how  could  knowledge  and 
skill  acquisition  be  confirmed  prior  to  selection? 

Supporting 

Reclassification 

For  selected  MOS,  what,  if  any,  technical  skill 
requirements  overlap? 

What  savings  are  achieved  through  reduced 
reclassification  training? 

Individual 
Performance 
Measures  and 
Standards 

What  types  of  individual  performance  standards 
do  Army  researchers,  trainers,  and  policy¬ 
makers  need? 

What  standard  setting  methodologies  would  be 
most  useful  for  helping  to  set  Army  individual 
performance  standards? 

Documenting  Unit 
Administrator  Task 
Requirements 

What  positions  require  RC-specific  analysis? 

What  are  the  responsibilities  of  an  RC  unit 
administrator? 

Management  Systems 

How  can  training  and  deployment  activities  be 
more  effectively  orchestrated? 

Can  some  aspects  of  assignments  and 
promotions  be  effectively  centralized? 

Are  ARPERSCOM  efforts  to  reorganize 
improving  organizational  mission  effectiveness? 

Leadership 

What  characteristics  make  leaders  successful  in 
the  RC  environment? 

What  tools  are  useful  for  leadership 
development  in  the  field? 

What  is  the  extent  and  impact  of  component 
biases  among  AC  and  RC  leaders? 
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generate  some  ideas  regarding  RC-related  research  priorities.  What  follows  in  this  section,  then,  is  a 
research  agenda  that  ARI  can  use  to  help  define  its  research  programs  in  the  coming  years. 

General  Research  Priorities 

The  critical  research  questions  identified  in  this  report,  by  design,  relate  to  research  areas  that 
are  either  within  the  scope  of  ARI’s  current  or  recent  research  programs  or  are  compatible  with  those 
programs  and  the  research  and  consulting  expertise  they  represent.  Table  10  depicts  how  these 
questions  relate  to  ARI’s  recent,  current,  and  future  research  agendas.  The  rows  in  Table  10  list  the 
research  domains,  and  the  areas  encompassed  by  each.  The  first  two  columns  indicate  which  areas 
have  been  addressed  with  significant  amounts  of  past,  present,  and  fiiture  ARI  research.  The  first 
column  identifies  areas  that  have  had  a  relatively  heavy  explicit  focus  on  the  RC  whereas  the  second 
column  identifies  areas  for  which  ARI  has  conducted  relevant  research  on  the  AC,  with  relatively  little 
specific  emphasis  on  the  RC.  The  identified  areas  in  these  columns  could  be  considered  the  “core 
business”  of  ARI’s  research  because  the  findings  of  this  project  suggest  that  ARI  work  in  these 
research  areas  should  continue. 

The  last  column  in  Table  10  indicates  the  areas  that  are  in  need  of  RC-specific  research 
attention  in  the  immediate  and  longer-term  future.  Every  research  area  is  marked,  reflecting  the 
fact  that  there  are  RC-relevant  research  needs  in  each.  The  column  includes  ARI’s  “core 
business”  areas  that  have  previously  been  targeted  toward  the  RC  (e.g.,  training  support,  DL), 
core  areas  that  have  previously  been  focused  primarily  on  the  AC  (e.g.,  TADSS-based  fidelity, 
recruiting),  and  areas  that  represent  a  bit  of  a  departure  from  ARI’s  previous  research  programs 
(e.g.,  management  systems).  We  have  assigned  rough  priorities  to  the  different  areas  as 
discussed  below. 

The  areas  related  to  training  that  have  previously  enjoyed  ARI  research  support,  we 
believe,  should  continue  to  get  the  highest  priority  in  ARI’s  future  RC-related  research  agenda. 
This  includes  areas,  such  as  TADSS  cost  effectiveness  tradeoffs,  that  can  continue  to  have  a 
broad  research  focus  (i.e.,  RC-specific  research  is  not  required),  but  which  likely  have  a  higher 
research  priority  for  the  RC  than  the  AC  because  the  RC  has  more  limited  training  resources. 
Both  the  broader  and  RC-specific  efforts  (e.g.,  on  DL  and  TADSS  strategies)  are  vital  for 
maintaining  a  close  relationship  with  RC  units  and  leadership.  Indeed,  one  of  the  RC  leaders  we 
interviewed  who  deals  with  TADSS-based  training  concluded  his  interview  with  the  comment: 
“ARI  has  a  real  role  to  play.  If  ARI  does  not  provide. .  .information,  where  will  I  get  it?"  (COL 
R.  Krug,  personal  communication,  December  12, 1996).  Such  positive  relationships  with  units 
and  leaders  will  keep  ARI  expertise  on  the  RC  environment  current  and  visible. 

Several  of  the  non-training  research  areas  marked  in  the  last  column  of  Table  10  have 
been  examined  in  ARI  research  that  has  not  previously  had  a  strong  explicit  RC  focus,  but  for 
which  RC-specific  work  is  needed.  For  example,  research  techniques  developed  to  study 
retention  and  turnover  in  the  AC  could  be  readily  adapted  for  similar  questions  in  the  RC.  These 
areas  should  be  moderate  to  high  priority  in  ARI’s  research  agenda  for  several  reasons.  For 
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Table  10 


ARI  Research  Agendas 


I 


Domain 


TADSS-Based 

Training 


Performance 

Measurement 


Training 

Management 


Distance 

Training 


Area 


Cost  Effectiveness  Tradeoffs 

Part-Task  Training _ 

TADSS-Based  Fidelity 
Computer-Generated  Forces 


Traimng  Support 


Strategy _ 

New  instructional  techniques 
and  research  strategies 


Diagnostic  Measures 
Proficiency  Assessment 


Macro-Level  Training  Strategy 

Database  Development  and 
Maintenance 


Distance  Learning 


Recent/Current 


RC  Focus  Broad  Focus 


X 


Future 


1* 


Distributed  Interactive 
Simulation 


Recruiting 


Retention/T  umover 


Selection,  Selecting  Enlisted  Personnel 

Classification,  with  Technical  Skills 

and 

Performance 

Management  _ 

Supporting  Reclassification 


Individual  Performance 
Measures  and  Standards 
Documenting  Unit 
Administrator  Task 
Requirements 


Management  Systems 


Leadership 


Note.  Entries  marked  with  an  asterisk  refer  to  research  that  may  be  conducted  with  the  AC  and  still  be  applicable  to 
RC  needs.  Numbers  reflect  rough  research  priorities. 
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example,  capitalizing  on  AC  prototypes  (e.g.,  adapting  the  CATS  calendar  to  the  RC)  will  reduce 
the  scope  of  projects  and  provide  a  high  expectation  of  success.  Also,  work  that  has  a  broader 
relevance  is  likely  to  make  greatest  use  of  prior  research  findings  and  has  a  higher  potential 
payoff  than  work  that  is  only  relevant  to  one  component  or  the  other.  This  includes  considering 
what  research  may  be  applicable  to  both  the  USAR  and  the  ARNG,  but  not  necessarily  the  AC. 

The  lowest  priority  areas  in  the  last  column  of  Table  10  are  those  that  begin  to  move 
beyond  ARI’s  current  core  business  areas.  These  areas  represent  needs  identified  by 
interviewees  and  that  are  well  within  the  capability  that  ARI  has  demonstrated.  While  these 
research  areas,  as  a  group  have  the  lowest  priority,  any  of  these  areas  might  be  given  high 
priority  by  RC  leaders.  If  that  happens,  ARI  is  well  positioned  in  terms  of  staff  expertise  to  add 
the  area  to  its  research  program. 

Other  Issues  Related  to  Research  Priorities 

There  are  several  other  key  issues  to  be  considered  in  determining  specific  research 
priorities.  Perhaps  most  importantly  is  the  issue  of  resources.  ARI  is  experiencing  large  cuts  in 
personnel  and  research  dollars  that  must  be  reconciled  with  Army  manpower,  personnel,  and 
training  research  needs.  Clearly,  not  all  research  needs  can  be  fully  addressed  in  such  an 
environment.  Making  the  most  of  current  resources  combined  with  leveraging  research  that  has 
demonstrated  cost-savings  to  the  Army  to  obtain  additional  resources  in  the  future  are  critical 
tasks  for  ARI. 

Another  issue  related  to  priorities  is  the  extent  to  which  applicable  research  is  being 
conducted  elsewhere  (e.g.,  under  the  sponsorship  of  DoD  or  one  of  the  other  Services). 
Coordinating  with  related  research  programs  to  reduce  costs  for  all  could  be  an  important 
strategy  for  addressing  some  Army  RC  research  needs.  Such  a  strategy  could  allow  ARI  to 
stretch  resources  to  address  some  areas  of  research  that  it  could  not  otherwise  include  in  its 
research  program. 

Looking  for  underlying  links  across  research  areas  is  one  more  key  to  determining 
research  priorities.  For  example,  work  with  the  recruiting  and  retention/tumover  domains  also 
offers  opportunities  for  expanding  RC  support  for  ARI  research.  Almost  all  leaders  interviewed 
considered  RC  manning  to  be  a  major  problem,  but  there  was  also  a  sense  that  manning  was 
largely  a  function  of  the  quality  of  TADSS  (which  was  thought  to  affect  recruiting)  and  quality 
of  training,  options  for  MOSQ,  and  quality  of  leadership  (which  were  thought  to  affect 
retention/tumover).  In  addition  to  addressing  questions  related  to  maiming  directly,  it  would  be 
desirable  to  treat  recraiting,  retention,  and  turnover  as  criterion  measures  for  assessing  the  impact 
of  the  other  research  areas. 
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SUMMARY 


In  this  final  section  we  offer  several  suggestions  in  an  effort  to  assist  ARI  in  using  the 
results  of  this  project  to  help  meet  the  research  needs  of  the  RC.  These  suggestions  involve  (a) 
commimications  with  sponsors,  (b)  interfaces  with  other  research  organizations,  (c)  using  this 
research  agenda  as  a  starting  point,  and  (d)  following  up  on  the  ideas  discussed  in  and  generated 
by  this  piece  of  work. 

Communications  with  Sponsors 

It  is  critical  that  ARI  establish  and  maintain  two-way  communications  with  its  current 
and  potential  research  sponsors.  There  is  already  a  well-established  relationship  with  ARNG 
training  elements,  but  relationships  with  other  ARNG  interests  and  the  USAR  need  to  be 
developed  and  nurtured.  Indeed,  the  USAR  personnel  we  interviewed  had  only  limited 
familiarity  with  ARI,  and  had  little  idea  of  how  to  approach  ARI  about  their  research  needs. 

Interfaces  with  Other  Research  Organizations 

By  establishing  interfaces  with  the  other  Services  and  selected  research  institutions  (e.g., 
IDA,  Rand,  Amerind)  who  are  engaged  in  related  work,  ARI  can  become  a  leader  in  fostering  the 
coordination  and  sharing  of  RC-related  research  programs  and  findings.  One  strategy  would  be 
to  sponsor  a  forum  for  sharing  ideas  and  research.  This  could  be  done  electronically  via,  for 
example,  an  Internet  bulletin  board  or  through  periodic  conferences.  Conferences  have  some 
benefits,  particularly  in  terms  of  providing  the  opportunity  for  face-to-face  exchanges,  but  may 
not  be  practical  in  terms  of  travel  budgets  and  the  resources  required  to  maintain  the  quality  of 
papers.  Bulletin  boards,  if  informative  and  well-maintained,  would  likely  achieve  many  of  the 
same  benefits  in  a  more  cost-effective  maimer.  Yet  another  strategy  is  to  explore  Joint  Service  or 
DoD  funding  of  research  of  interest  across  one  or  more  of  the  Services.  This  strategy  has  been 
successfully  used  in  the  past  (e.g.,  the  Joint-Service  classification  research  roadmap)  and  helps  to 
assuage  those  who  are  argue  that  the  Services  approach  research  problems  too  unilaterally.  On  a 
related  note,  RC  research  should  be  explicitly  considered  in  the  development  of  Joint-Service 
research  initiatives  as  defined  by  the  TABSTEM  group. 

A  Starting  Point 

The  research  agenda  discussed  in  this  report  should  be  viewed  as  a  starting  point  rather 
than  an  inflexible  prescription  for  future  research.  It  should  be  expected  to  evolve  as  issues  and 
resources  change  over  time,  and  close  lines  of  commimication  with  research  sponsors  will  help  to 
assure  this  evolution  takes  place.  Moreover,  it  will  be  left  to  ARI  and  its  sponsors  to  draw  from 
the  suggestions  provided  herein  and,  combined  with  their  imderstanding  of  resource  constraints 
and  additional  relevant  information  (e.g.,  leader  priorities,  Congressionally-imposed 
requirements),  to  translate  this  research  agenda  into  a  specific  plan  of  action  for  ARI. 
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Follow-Up 


The  two  volumes  produced  by  this  project  will  not  accomplish  anything  in  and  of 
themselves.  Both  ARI  and  RC  readers  need  to  take  responsibility  for  discussing  their  ideas  and 
reactions  to  this  work  with  those  who  will  be  responsible  for  obtaining  and  obligating  ARI 
research  resources  in  the  future. 
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APPENDIX  A 


PROJECT  INTERVIEWS 


Tables  A-1  and  A-2  provide  a  summary  of  the  individuals  who  participated  in  project 
interviews.  Our  goal  was  to  interview  researchers  and  RC  personnel  representing  a  variety  of 
perspectives,  including  those  most  familiar  with  training,  manpower,  or  persoimel  issues  and 
individuals  at  higher  levels  who  are  responsible  for  research  and  operations  that  span  all  of  these 
areas.  Table  A-1  lists  the  RC  persoimel  from  both  the  USAR  and  the  ARNG  who  were  included  in 
the  interviews.  Current  and  former  ARI  researchers  were  interviewed,  as  well  as  a  small  number  of 
knowledgeable  researchers  from  other  organizations.  They  are  listed  in  Table  A-2. 

At  least  one  of  the  two  senior  researchers  on  this  project  participated  in  each  interview. 
Responsible  ARI  researchers  participated  in  many  of  the  interviews  as  well.  This  was  particularly 
desirable  because  the  interviews  served  as  a  valuable  ARI  marketing  initiative  in  and  of  themselves. 
That  is,  the  face-to-face  contact  with  a  variety  of  current  and  potential  clients  was  an  important  step 
in  expanding  and  nurturing  ARI’s  contact  with  the  RC  part  of  its  user  community. 

The  interviews  were  semi-structured.  We  introduced  all  interviewees  to  the  project  by 
describing  to  them  the  project  goals,  the  approach,  and  the  purpose  of  the  interviews.  We  then 
asked  interviewees  to  discuss  trends  affecting  the  RC  and  to  describe  research  areas  that  they 
believe  to  be  high  priority  for  the  current  and  future  RC.  Interviewees  were  also  asked  to  provide 
any  documents  or  other  materials  that  might  be  useful  to  us  in  the  development  of  an  RC 
research  roadmap. 


A-1 


Table  A-1 


Summaiy  of  Project  Interviews  with  RC  Personnel 


Date 

Intervievyee(s) 

Representing 

1  Nov  96 

BG  Helmly,  COL  Guild,  LTC  Joel, 
COL  McCloskey,  MAJ  Egbert,  Ms. 
Kelly,  LTC  Lerch 

Office  of  the  Chief  of  Staff  of  the  Army 

Reserve  (OCAR);  Personnel,  Budget,  Strategic 
Analysis,  PAE 

18  Nov  96 

MAJ  Egbert 

OCAR,  Office  of  Strategic  Analysis  &  Liaison 

18  Nov  96 

LTC  Fields,  MAJ  Ford 

OCAR,  Training 

26  Nov  96 

LTC  Joel,  MAJ  Meyer,  LTC 
Westmoreland 

OCAR,  Personnel 

28  Oct  96 

COL  Squier 

Chief  of  Staff 

12  Nov  96 

LTC  Martinez,  CPT  Boston 

NGB,  Army  Research  Office;  SI  of  deployed 
unit 

18  Nov  96 

COL  Himle,  LTC  Otto 

NGB,  Training,  Training  Support 

12  Dec  96 

COL  Youngman,  LTC  Myers 

NGB;  KY  ARNG,  Brigade  Commander  and 

NGB  Research  and  Staff  Support  Office 

12  Dec  96 

COL  Krug 

Advanced  Research  Projects  Agency, 

Information  Systems  Office  Program  Manager 

17  Dec  96 

BG  Groves,  COL  Smith,  COL  Jones, 
Mr.  Barker 

KYARNG:  The  Adjutant  General,  Chief  of 
Staff,  US  Property  Fiscal  Officer,  Dept,  of 
Military  Affairs 

17  Dec  96 

COL  Nelson,  COL  Sullivan,  COL 
Carpenter 

KY  ARNG:  Senior  Army  Advisor;  Inspector 
General;  Plans,  Operations,  and  Training 

17  Dec  96 

COL  Gilbert,  COL  Shelley 

KY  ARNG;  Surface  Maintenance,  Command 
Logistics 

17  Dec  96 

LTC  Sebastian,  LTC  Waggoner,  LTC 
Storm 

KY  ARNG:  Human  Resources,  Recruiting  & 
Retention,  Personnel 

17  Dec  96 

LTC  Hetzel 

KY  ARNG:  Director  of  Information 

Management 

17  Dec  96 

LTC  Fincham,  LTC  Wadley 

KY  ARNG:  Construction  Facilities 

Management,  Safety  and  Occupational  Health 

7  Feb  97 

COL  Gordon 

ID  ARNG:  Senior  AC  Advisor 

7  Feb  97 

COL  La  Frenz 

ID  ARNG:  Commander,  1 16th  Calvary 

Brigade 

7  Feb  97 

MG  Kane 

ID  ARNG:  The  Adjutant  General 

A-2 


Table  A-2 


Summary  of  Project  Interviews  with  Researchers 


Date 

Interviewee(s) 

Representing 

8  Oct  96 

Drs.  Gade,  Seidel,  Rumsey,  Peterson, 
Drillings 

Personnel  Resources,  Training,  Selection 
&  Assignment,  Surveys,  Basic  Research 

17  Oct  96 

Dr.  Michael  Rumsey 

Selection  &  Assignment 

17  Oct  96 

Dr.  Alma  Steinberg 

Trend  Analysis 

23  Oct  96 

Dr.  Mike  Sanders 

Special  Operations  Forces 

24  Oct  96 

Dr.  Trueman  Tremble 

Leadership 

25  Oct  96 

Dr.  Robert  Seidel 

Training 

25  Oct  96 

Dr.  Darrell  Wostine 

Occupational  Analysis 

15  Nov  96 

Dr.  Robert  Sulzen 

Training 

18  Nov  96 

Dr.  Scott  Graham 

Training  (Infantry) 

22  Nov  96 

Dr.  Steve  Goldberg 

Training 

8  Jan  97 

Dr.  Stan  Halpin 

Leadership 

10  Feb  97 

Dr.  Robert  Wisher 

Training 

13  Dec  96 

Drs.  Dexter  Fletcher,  John  Metzko, 

John  Morrison,  Jesse  Orlansky 

Institute  for  Defense  Analysis 

3  Jan  97 

Dr.  Glenda  Nogam  i 

US  Army  War  College,  former  ARI 
researcher 

3  Jan  97 

Dr.  Curt  Gilroy 

Former  ARI  researcher 

A-3 


